Vector Supercomputers Research Articles

Conflict-aware workload co-execution on SX-aurora TSUBASA

NEC SX-Aurora TSUBASA (SX-AT) is the latest vector supercomputer, consisting of host processors called Vector Hosts (VHs) and vector processors called Vector Engines (VEs). The goal of this work is to simultaneously use both VHs and VEs to increase the resource utilization and improve the system throughput by co-executing more workloads. One difficulty is that performance interferences among VH and VE workloads could occur because they share some computing resources and potentially compete to use the same resource at the same time, so-called resource conflicts. To achieve efficient workload co-execution, first, this paper experimentally investigates the performance interference between a VH and a VE, when each of the two processors executes a different workload. It is empirically shown that the frequency of system calls from the VE workload could be a good indicator to predict if the co-execution could cause severe performance interference, even though monitoring system calls requires a huge runtime overhead and it is impractical to simply use it for decision making of co-execution. Then, this paper proposes a workload co-execution strategy based on a practical approach to identifying a pair of VE and VH workloads that could cause severe performance interferences. Our evaluation results clearly demonstrate that the system call frequency can be used to predict if the workload can affect the performance of another co-executing workload, and VH’s CPU load can be a good approximation of the system call frequency. The proposed approach based on the CPU loads could accurately identify a pair of workloads causing frequent resource conflicts, and thus reduce the risk of severe performance interferences between co-executing workloads on an SX-AT system, resulting in shorter makespan without significantly increasing the turn-around time.

Optimizing Load Balance in a Parallel CFD Code for a Large-scale Turbine Simulation on a Vector Supercomputer

A turbine for power generation is one of the essential infrastructures in our society. A turbine's failure causes severe social and economic impacts on our everyday life. Therefore, it is necessary to foresee such failures in advance. However, it is not easy to expect these failures from a real turbine. Hence, it is required to simulate various events occurring in the turbine by numerical simulations of the turbine. A multiphysics CFD code, ‘‘Numerical Turbine,’' has been developed on vector supercomputer systems for large-scale simulations of unsteady wet steam flows inside a turbine. To solve this problem, the Numerical Turbine code is a block structure code using MPI parallelization, and the calculation space consists of grid blocks of different sizes. Therefore, load imbalance occurs when executing the code in MPI parallelization. This paper creates an estimation model that finds the calculation time from each grid block's calculation amount and calculation performance. It proposes an OpenMP parallelization method for the load balance of MPI applications. This proposed method reduces the load imbalance by considering the vector performance according to the calculation amount based on the model. Moreover, this proposed method recognizes the need to reduce the load imbalance without pre-execution. The performance evaluation shows that the proposed method improves the load balance from 24.4 % to 9.3 %.

VGL: a high-performance graph processing framework for the NEC SX-Aurora TSUBASA vector architecture

Developing efficient graph algorithms implementations is an extremely important problem of modern computer science, since graphs are frequently used in various real-world applications. Graph algorithms typically belong to the data-intensive class, and thus using architectures with high-bandwidth memory potentially allows to solve many graph problems significantly faster compared to modern multicore CPUs. Among other supercomputer architectures, vector systems, such as the SX family of NEC vector supercomputers, are equipped with high-bandwidth memory. However, the highly irregular structure of many real-world graphs makes it extremely challenging to implement graph algorithms on vector systems, since these implementations are usually bulky and complicated, and a deep understanding of vector architectures hardware features is required. This paper presents the world first attempt to develop an efficient and simultaneously simple graph processing framework for modern vector systems. Our vector graph library (VGL) framework targets NEC SX-Aurora TSUBASA as a primary vector architecture and provides relatively simple computational and data abstractions. These abstractions incorporate many vector-oriented optimization strategies into a high-level programming model, allowing quick implementation of new graph algorithms with a small amount of code and minimal knowledge about features of vector systems. In this paper, we evaluate the VGL performance on four widely used graph processing problems: breadth-first search, single source shortest paths, connected components, and page rank. The provided comparative performance analysis demonstrates that the VGL-based implementations achieve significant acceleration over the existing high-performance frameworks and libraries: up to 14 times speedup over multicore CPUs (Ligra, Galois, GAPBS) and up to 3 times speedup compared to NVIDIA GPU (Gunrock, NVGRAPH) implementations.

Effects of Inflow Shear on Wake Characteristics of Wind-Turbines over Flat Terrain

The scope of the present study was to understand the wake characteristics of wind-turbines under various inflow shears. First, in order to verify the prediction accuracy of the in-house large-eddy simulation (LES) solver, called RIAM-COMPACT, based on a Cartesian staggered grid, we conducted a wind-tunnel experiment using a wind-turbine scale model and compared the numerical and experimental results. The total number of grid points in the computational domain was about 235 million. Parallel computation based on a hybrid LES/actuator line (AL) model approach was performed with a new SX-Aurora TSUBASA vector supercomputer. The comparison between wind-tunnel experiment and high-resolution LES results showed that the AL model implemented in the in-house LES solver in this study could accurately reproduce both performances of the wind-turbine scale model and flow characteristics in the wake region. Next, with the LES solver developed in-house, flow past the entire wind-turbine, including the nacelle and the tower, was simulated for a tip-speed ratio (TSR) of 4, the optimal TSR. Three types of inflow shear, N = 4, N = 10, and uniform flow, were set at the inflow boundary. In these calculations, the calculation domain in the streamwise direction was very long, 30.0 D (D being the wind-turbine rotor diameter) from the center of the wind-turbine hub. Long-term integration of t = 0 to 400 R/Uin was performed. Various turbulence statistics were calculated at t = 200 to 400 R/Uin. Here, R is the wind-turbine rotor radius, and Uin is the wind speed at the hub-center height. On the basis of the obtained results, we numerically investigated the effects of inflow shear on the wake characteristics of wind-turbines over a flat terrain. Focusing on the center of the wind-turbine hub, all results showed almost the same behavior regardless of the difference in the three types of inflow shear.

Development and Validation of a Tsunami Numerical Model with the Polygonally Nested Grid System and its MPI-Parallelization for Real-Time Tsunami Inundation Forecast on a Regional Scale

We have developed a new numerical model suitable for rapid and wide-area estimation of tsunami inundation and damage. The model is based on the world-renowned TUNAMI code solving the two-dimensional nonlinear shallow water equations, and enables one-stop simulation of the initial tsunami distribution based on a fault model, tsunami propagation and inundation, and damage estimation. It extends the configuration of the grid system from conventional rectangular regions to polygonal regions so that deployment of high-resolution grids can be confined to the coastal lowland, resulting in remarkably improved efficiency in computation and better precision. For the purpose of real-time implementation of tsunami inundation simulation using a high-performance computing infrastructure, vectorization and MPI parallelization have also been conducted. Moreover, the model was verified and validated through several benchmark problems that the National Tsunami Hazard Mitigation Program, organized by federal agencies and states in the U.S., developed as the quality standards for simulating and assessing tsunami hazard and risk. The newly-developed model is named “Real-time Tsunami inundation (RTi) model,” and its computational performance was examined using the SX-ACE, a vector supercomputer installed at Tohoku University. The results show that it requires only 128 cores of the SX-ACE for implementing six-hour tsunami inundation simulation with a 10-meter grid resolution within 10 minutes for the 700 km long coastline of Kochi Prefecture, Japan. This means that the RTi model is over 10 times more efficient as the conventional tsunami model with the rectangular domains, and it can be inferred that 2,451 cores of the SX-ACE are the overall computational resources needed for real-time tsunami inundation forecast on the whole coastal regions along the Nankai Trough subduction zone, corresponding to the computational performance of 170 Tflop/s. The resources required are equivalent to 24% of all the SX-ACE resources at Tohoku University, indicating the feasibility of real-time tsunami inundation forecast on a regional scale by using the RTi model. Since the Disaster Information System operated by the Cabinet Office of the Japanese Government adopted a function of tsunami damage estimation using the aforementioned numerical model, at the end of this paper, a brief overview of the subsystem for rapidly estimating tsunami damage on a regional scale is described.

Performance Evaluation of Different Implementation Schemes of an Iterative Flow Solver on Modern Vector Machines

Modern supercomputers consist of multi-core processors, and these processors have recently employed vector instructions, or so-called SIMD instructions, to improve performances. Numerical simulations need to be vectorized in order to achieve higher performance on these processors. Various legacy numerical simulation codes that have been utilized for a long time often contain two versions of source codes: a non-vectorized version and a vectorized version that is optimized for old vector supercomputers. It is important to clarify which version is better for modern supercomputers in order to achieve higher performance. In this paper, we evaluate the performances of a legacy fluid dynamics simulation code called FASTEST on modern supercomputers in order to provide a guidepost for migrating such codes to modern supercomputers. The solver has a nonvectorized version and a vectorized version, and the latter uses the hyperplane ordering method for vectorization. For the evaluation, we also implement the red-black ordering method, which is another way to vectorize the solver. Then, we examine the performance on NEC SX-ACE, SXAurora TSUBASA, Intel Xeon Gold, and Xeon Phi. The results show that the shortest execution times are with the red-black ordering method on SX-ACE and SX-Aurora TSUBASA, and with the non-vectorized version on Xeon Gold and Xeon Phi. Therefore, achieving a higher performance on multiple modern supercomputers potentially requires maintenance of multiple code versions. We also show that the red-black ordering method is more promising to achieve high performance on modern supercomputers.

Mixed Precision: A Strategy for New Science Opportunities

Since the days of vector supercomputers, computational scientists have relied on high-precision arithmetic to accurately solve problems. But changes to hardware, spurred by the demand for more computing capability and growth in machine learning, have researchers considering lower precision alternatives.

Real-time tsunami inundation forecast system for tsunami disaster prevention and mitigation

The tsunami disasters that occurred in Indonesia, Chile, and Japan have inflicted serious casualties and damaged social infrastructures. Tsunami forecasting systems are thus urgently required worldwide. We have developed a real-time tsunami inundation forecast system that can complete a tsunami inundation and damage forecast for coastal cities at the level of 10-m grid size in less than 20 min. As the tsunami inundation and damage simulation is a vectorizable memory-intensive program, we incorporate NEC’s vector supercomputer SX-ACE. In this paper, we present an overview of our system. In addition, we describe an implementation of the program on SX-ACE and evaluate its performance of SX-ACE in comparison with the cases using an Intel Xeon-based system and the K computer. Then, we clarify that the fulfillment of a real-time tsunami inundation forecast system requires a system with high-performance cores connected to the memory subsystem at a high memory bandwidth such as SX-ACE.

A Real-Time Tsunami Inundation Forecast System Using Vector Supercomputer SX-ACE

Tsunami disasters can cause serious casualties and damage to social infrastructures. An early understanding of disaster states is required in order to advise evacuations and plan rescues and recoveries. We have developed a real-time tsunami inundation forecast system using a vector supercomputer SX-ACE. The system can complete a tsunami inundation and damage estimation for coastal city regions at the resolution of a 10 m grid size in under 20 minutes, and distribute tsunami inundation and infrastructure damage information to local governments in Japan. We also develop a new configuration for the computational domain, which is changed from rectangles to polygons and called a polygonal domain, in order to effectively simulate in the entire coast of Japan. Meanwhile, new supercomputers have been developed, and their peak performances have increased year by year. In 2016, a new Xeon Phi processor calledKnights Landingwas released for high-performance computing. In this paper, we present an overview of our real-time tsunami inundation forecast system and the polygonal domain, which can decrease the amount of computation in a simulation, and then discuss its performance on a vector supercomputer SX-ACE and a supercomputer system based on Intel Xeon Phi. We also clarify that the real-time tsunami inundation forecast system requires the efficient vector processing of a supercomputer with high-performance cores.

Dynamical Downscaling of SINTEX-F2v CGCM Seasonal Retrospective Austral Summer Forecasts over Australia

An ensemble of 1-month-lead seasonal retrospective forecasts generated by the Scale Interaction Experiment (SINTEX)–Frontier Research Center for Global Change (FRCGC), version 2 tuned for performance on a vector supercomputer (SINTEX-F2v), coupled global circulation model (CGCM) were downscaled using the Weather Research and Forecasting (WRF) Model to improve the forecast of the austral summer precipitation and 2-m air temperatures over Australia. A set of four experiments was carried out with the WRF Model to improve the forecasts. The first was to drive the WRF Model with the SINTEX-F2v output, and the second was to bias correct the mean component of the SINTEX-F2v forecast and drive the WRF Model with the corrected fields. The other experiments were to use the SINTEX-F2v forecasts and the mean bias-corrected SINTEX-F2v forecasts to drive the WRF Model coupled to a simple mixed layer ocean model. Evaluation of the forecasts revealed the WRF Model driven by bias-corrected SINTEX-F2v forecasts to have a better spatial and temporal representation of forecast precipitation and 2-m air temperature, compared to SINTEX-F2v forecasts. Using a regional coupled model with the bias-corrected SINTEX-F2v forecast as the driver further improved the skill of the precipitation forecasts. The improvement in the WRF Model forecasts is due to better representation of the variables in the bias-corrected SINTEX-F2v forecasts driving the WRF Model. The study brings out the importance of including air–sea interactions and correcting the global forecasts for systematic biases before downscaling them for societal applications over Australia. These results are important for potentially improving austral summer seasonal forecasts over Australia.

ベクトル計算機におけるFill-in付き(M)ICCG法の性能評価

ベクトル計算機におけるFill-in付き(M)ICCG法の性能評価

Potential of a modern vector supercomputer for practical applications: performance evaluation of SX-ACE

Achieving a high sustained simulation performance is the most important concern in the HPC community. To this end, many kinds of HPC system architectures have been proposed, and the diversity of the HPC systems grows rapidly. Under this circumstance, a vector-parallel supercomputer SX-ACE has been designed to achieve a high sustained performance of memory-intensive applications by providing a high memory bandwidth commensurate with its high computational capability. This paper examines the potential of the modern vector-parallel supercomputer through the performance evaluation of SX-ACE using practical engineering and scientific applications. To improve the sustained simulation performances of practical applications, SX-ACE adopts an advanced memory subsystem with several new architectural features. This paper discusses how these features, such as MSHR, a large on-chip memory, and novel vector processing mechanisms, are beneficial to achieve a high sustained performance for large-scale engineering and scientific simulations. Evaluation results clearly indicate that the high sustained memory performance per core enables the modern vector supercomputer to achieve outstanding performances that are unreachable by simply increasing the number of fine-grain scalar processor cores. This paper also discusses the performance of the HPCG benchmark to evaluate the potentials of supercomputers with balanced memory and computational performance against heterogeneous and cutting-edge scalar parallel systems.

Parallel-vector algorithms for particle simulations on shared-memory multiprocessors

Over the last few decades, the computational demands of massive particle-based simulations for both scientific and industrial purposes have been continuously increasing. Hence, considerable efforts are being made to develop parallel computing techniques on various platforms. In such simulations, particles freely move within a given space, and so on a distributed-memory system, load balancing, i.e., assigning an equal number of particles to each processor, is not guaranteed. However, shared-memory systems achieve better load balancing for particle models, but suffer from the intrinsic drawback of memory access competition, particularly during (1) paring of contact candidates from among neighboring particles and (2) force summation for each particle. Here, novel algorithms are proposed to overcome these two problems. For the first problem, the key is a pre-conditioning process during which particle labels are sorted by a cell label in the domain to which the particles belong. Then, a list of contact candidates is constructed by pairing the sorted particle labels. For the latter problem, a table comprising the list indexes of the contact candidate pairs is created and used to sum the contact forces acting on each particle for all contacts according to Newton’s third law. With just these methods, memory access competition is avoided without additional redundant procedures. The parallel efficiency and compatibility of these two algorithms were evaluated in discrete element method (DEM) simulations on four types of shared-memory parallel computers: a multicore multiprocessor computer, scalar supercomputer, vector supercomputer, and graphics processing unit. The computational efficiency of a DEM code was found to be drastically improved with our algorithms on all but the scalar supercomputer. Thus, the developed parallel algorithms are useful on shared-memory parallel computers with sufficient memory bandwidth.

Characteristics of an On-Chip Cache on NEC SX Vector Architecture

Thanks to the highly effective memory bandwidth of the vector systems, they can achieve the high computation efficiency for computation-intensive scientific applications. However, they have been encountering the memory wall problem and the effective memory bandwidth rate has decreased, resulting in the decrease in the bytes per flop rates of recent vector systems from 4 (SX-7 and SX-8) to 2 (SX-8R) and 2.5 (SX-9). The situation is getting worse as many functions units and/or cores will be brought into a single chip, because the pin bandwidth is limited and does not scale. To solve the problem, we propose an on-chip cache, called vector cache, to maintain the effective memory bandwidth rate of future vector supercomputers. The vector cache employs a bypass mechanism between the main memory and register files under software controls. We evaluate the performance of the vector cache on the NEC SX vector processor architecture with bytes per flop rates of 2 B/FLOP and 1 B/FLOP, to clarify the basic characteristics of the vector cache. For the evaluation, we use the NEC SX-7 simulator extended with the vector cache mechanism. Benchmark programs for performance evaluation are two DAXPY-like loops and five leading scientific applications. The results indicate that the vector cache boosts the computational efficiencies of the 2 B/FLOP and 1 B/FLOP systems up to the level of the 4 B/FLOP system. Especially, in the case where cache hit rates exceed 50%, the 2 B/FLOP system can achieve a performance comparable to the 4 B/ FLOP system. The vector cache with the bypass mechanism can provide the data both from the main memory and the cache simultaneously. In addition, from the viewpoints of designing the cache, we investigate the impact of cache associativity on the cache hit rate, and the relationship between cache latency and the performance. The results also suggest that the associativity hardly affects the cache hit rate, and the effects of the cache latency depend on the vector loop length of applications. The cache shorter latency contributes to the performance improvement of the applications with shorter loop lengths, even in the case of the 4 B/FLOP system. In the case of longer loop lengths of 256 or more, the latency can effectively be hidden, and the performance is not sensitive to the cache latency. Finally, we discuss the effects of selective caching using the bypass mechanism and loop unrolling on the vector cache performance for the scientific applications. The selective caching is effective for efficient use of the limited cache capacity. The loop unrolling is also effective for the improvement of performance, resulting in a synergistic effect with caching. However, there are exceptional cases; the loop unrolling worsens the cache hit rate due to an increase in the working space to process the unrolled loops over the cache. In this case, an increase in the cache miss rate cancels the gain obtained by unrolling.

Optimal broadcast for fully connected processor-node networks

We develop and implement an optimal broadcast algorithm for fully connected processor networks under a bidirectional communication model in which each processor can simultaneously send a message to one processor and receive a message from another, possibly different processor. For any number of processors p the algorithm requires N − 1 + ⌈ log p ⌉ communication rounds to broadcast N blocks of data from a root processor to the remaining processors, meeting the lower bound in the model. For data of size m , assuming that sending and receiving data of size m ′ takes time α + β m ′ , the best running time that can be achieved by the division of m into equal-sized blocks is ( ( ⌈ log p ⌉ − 1 ) α + β m ) 2 . The algorithm uses a regular, circulant graph communication pattern, and degenerates into a binomial tree broadcast when the number of blocks to be broadcast is one. The algorithm is furthermore well suited to fully connected clusters of SMP ( Symmetric Multi-Processor) nodes. The algorithm is implemented as part of an MPI ( Message Passing Interface) library. We demonstrate significant practical bandwidth improvements of up to a factor 1.5 over several other, commonly used broadcast algorithms on both a small SMP cluster and a 72 node NEC SX vector supercomputer.

Interactive Simulation in Virtual Environments - A Design Tool for Planners and Architects

Simulations can assist planners in optimizing their design and in minimizing its environmental impact. By adjusting the architecture according to simulation results, and running further simulations based on the adjusted design, an iterative process can help to increase the design quality. Up to now computing simulations have taken a long time, thus only a very limited number of iterations could be calculated. This project shows an approach that is close to a real time simulation. By dividing the simulation into smaller parts and running the software on clusters or vector supercomputers, first results are available within several seconds, and reasonable results in less than one minute. Besides the technical features, another focus is the easy accessibility of the simulation. Intuitive methods like a tangible user interface provide easy interaction methods for specialists as well as non specialists. The results of the simulation can be visualized and interacted with from the desktop or any kind of virtual environments. Further aspects like limitations of automatic grid generation, shape recognition and computation power are discussed.

High performance computing technology, applications and business

High Performance Computing (HPC) was born in the mid 70’s with the emergence of vector supercomputers. It has then evolved according to technology and business enlarging progressively its scope of application. In this paper, we describe the fundamental concepts at the core of HPC, their evolution, the way they are used today in real applications, how these applications are evolving and how application and technology are transforming business.

20503 稠密炉心内3次元気泡流構造に関する大規模シミュレーション(大規模数値解析(1),OS11 大規模数値解析)

In order to predict the water-vapor two-phase flow structure in a fuel bundle of an advanced light water reactor, a large-scale numerical simulation was carried out using a newly developed two-phase flow analysis code and a highly parallel vector supercomputer. Conventional analysis methods with a two-fluid model need composition equations and empirical correlations based on the experimental data. Therefore, it is difficult to obtain high prediction accuracy when experimental data are nothing. Then, a new two-phase flow analysis method based on the latest computational science was proposed. This paper describes the predicted bubbly flow dynamics in three-dimensional minichannels and the predicted void fractions in the simulated tight-lattice fuel channels. Regarding the coalescence and fragmentation of bubbles the useful knowledge was obtained, and also the high prospect was acquired on the possibility of development of the thermal design procedure of the advanced nuclear reactors with large-scale simulations.

EARTH SIMULATOR AND SIMULATION RESEARCH PROGRESS

Earth simulator is a parallel vector super-computer which developed by National Space Development Agency of Japan(NASDA)and Japan Atomic Energy Research Institute(JAERI)and Japan Agency for Marine-Earth Science and Technology(JAMSTEC).Understanding and forecasting high-impact phenomena in the atmosphere and ocean,global warm,crust.The performance of Earth Simulator,birth of the Earth Simulator,research contents of five groups which are Atmosphere and Ocean Simulation Research Group,Solid Earth Simulation Research Group,Multiscale Simulation Research Group,Advanced Perception Research Group and Holistic Simulation Research Group,research projects and collaborations are introduced.At last,usage details details of the Earth Simulator and sustained performance of actual application are introduced.

Hardware system of the Earth Simulator

The Earth Simulator (ES), developed under the Japanese government’s initiative “Earth Simulator project”, is a highly parallel vector supercomputer system. In this paper, an overview of ES, its architectural features, hardware technology and the result of performance evaluation are described. In May 2002, the ES was acknowledged to be the most powerful computer in the world: 35.86 teraflop/s for the LINPACK HPC benchmark and 26.58 teraflop/s for an atmospheric general circulation code (AFES). Such a remarkable performance may be attributed to the following three architectural features; vector processor, shared-memory and high-bandwidth non-blocking interconnection crossbar network. The ES consists of 640 processor nodes (PN) and an interconnection network (IN), which are housed in 320 PN cabinets and 65 IN cabinets. The ES is installed in a specially designed building, 65 m long, 50 m wide and 17 m high. In order to accomplish this advanced system, many kinds of hardware technologies have been developed, such as a high-density and high-frequency LSI, a high-frequency signal transmission, a high-density packaging, and a high-efficiency cooling and power supply system with low noise so as to reduce whole volume of the ES and total power consumption. For highly parallel processing, a special synchronization means connecting all nodes, Global Barrier Counter (GBC), has been introduced.