NEC SX-8 Research Articles

A Memory-Efficient Implementation of a Plasmonics Simulation Application on SX-ACE

Since recent scientific and engineering simulations require heavy computations with large volumes of data, High-performance Computing (HPC) systems need a high computational capability with a large memory capacity. Most recent HPC systems adopt a parallel processing architecture, where the computational capability of the processors is increasing, however, the performance of the memory system is constrained. The bytes per flop (B/F), which is a ratio of the memory bandwidth to the flop/s, for the HPC systems have been reduced with the evolution of the HPC systems. To fully exploit the potential of the recent HPC systems, and to meet the increasing demand for large memory, it is necessary to optimize practical scientific and engineering applications, considering not only the parallelism of the applications, but also the limitations of the memory subsystems of the HPC systems. In this paper, we discuss a set of approaches to optimization of the memory access behavior of the applications, which enable their executions with improved performance on the recent HPC systems. Our approaches include memory optimizations through memory footprint controlling, restructuring of data structures for active elements, redundant data structure elimination through combined calculations and optimized re-calculation of data. To validate the effectiveness of our approaches, a plasmonics simulation application is evaluated on vector platforms NEC SX-ACE, NEC SX-9, and Intel Xeon based platform NEC LX 406-Re2. By applying our approaches to the implementation, the memory usage of the plasmonics simulation application can be reduced up to nearly 1/71 of the original, and its execution can be possible on a single node of a distributed parallel system with smaller memory capacity. The optimization results in 1.14 times faster execution on SX-ACE and 1.81 times faster execution on LX 406-Re2.

Efficient performance of the Met Office Unified Model v8.2 on Intel Xeon partially used nodes

Abstract. The atmospheric Unified Model (UM) developed at the UK Met Office is used for weather and climate prediction by forecast teams at a number of international meteorological centres and research institutes on a wide variety of hardware and software environments. Over its 25 year history the UM sources have been optimised for better application performance on a number of High Performance Computing (HPC) systems including NEC SX vector architecture systems and recently the IBM Power6/Power7 platforms. Understanding the influence of the compiler flags, Message Passing Interface (MPI) libraries and run configurations is crucial to achieving the shortest elapsed times for a UM application on any particular HPC system. These aspects are very important for applications that must run within operational time frames. Driving the current study is the HPC industry trend since 1980 for processor arithmetic performance to increase at a faster rate than memory bandwidth. This gap has been growing especially fast for multicore processors in the past 10 years and it can have significant implication for the performance and performance scaling of memory bandwidth intensive applications, such as the UM. Analysis of partially used nodes on Intel Xeon clusters is provided in this paper for short- and medium-range weather forecasting systems using global and limited-area configurations. It is shown that on the Intel Xeon-based clusters the fastest elapsed times and the most efficient system usage can be achieved using partially committed nodes.

Performance of SOR methods on modern vector and scalar processors

The building-cube method (BCM) is a new generation algorithm for CFD simulations. The basic idea of BCM is to simplify the algorithm in all stages of flow computation to achieve large-scale simulations. Calculation of a pressure field using the Successive Over Relaxation (SOR) method consumes most of the total execution time required for BCM. In this paper, effective implementations on modern vector and scalar processors are investigated. NEC SX-9 and Intel Nehalem-EX are the latest vector and scalar processors. Those processors have much higher peak performances than their previous-generation processors. However, their memory bandwidth improvement cannot catch up with the performance improvement of processors. This is the so-called memory wall problem. In our paper, we discuss optimization techniques for implementation of the SOR method based on architectural characteristics of these modern processors, and evaluate their effects on the sustained performances of these processors for BCM.

FEAST—realization of hardware‐oriented numerics for HPC simulations with finite elements

AbstractFEAST (Finite Element Analysis and Solutions Tools) is a Finite Element‐based solver toolkit for the simulation of PDE problems on parallel HPC systems, which implements the concept of ‘hardware‐oriented numerics’, a holistic approach aiming at optimal performance for modern numerics. In this paper, we describe this concept and the modular design that enables applications built on top of FEAST to execute efficiently, without any code modifications, on commodity‐based clusters, the NEC SX 8 and GPU‐accelerated clusters. We demonstrate good performance and weak and strong scalability for the prototypical Poisson problem and more challenging applications from solid mechanics and fluid dynamics. Copyright © 2010 John Wiley & Sons, Ltd.

A Pipelined Algorithm for Large, Irregular All-Gather Problems

We describe and evaluate a new pipelined algorithm for large, irregular all-gather problems. In the irregular allgather problem each process in a set of processes contributes individual data of possibly different size, and all processes have to collect all data from all processes. The pipelined algorithm is useful for the implementation of the MPI_Allgatherv collective operation of the Message-Passing Interface (MPI) for large problems. By conception, the new algorithm is well suited to implementation on clustered multiprocessors, such as symmetric multiprocessing (SMP) clusters. The new algorithm has been implemented within different MPI libraries. Benchmark results on NEC SX-8, Linux clusters with InfiniBand and Gigabit Ethernet, IBM Blue Gene/P, and SiCortex systems show huge performance gains in accordance with the expected behavior.

BENCHMARK ANALYSIS AND APPLICATION RESULTS FOR LATTICE BOLTZMANN SIMULATIONS ON NEC SX VECTOR AND INTEL NEHALEM SYSTEMS

Classic vector systems have all but vanished from recent TOP500 lists. Looking at the recently introduced NEC SX-9 series, we benchmark its memory subsystem using the low level vector triad and employ the kernel of an advanced lattice Boltzmann flow solver to demonstrate that classic vectors still combine excellent performance with a well-established optimization approach. To investigate the multi-node performance, the flow field in a real porous medium is simulated using the hybrid MPI/OpenMP parallel ILBDC lattice Boltzmann application code. Results for a commodity Intel Nehalem-based cluster are provided for comparison. Clusters can keep up with the vector systems, however, require massive parallelism and thus much more effort to provide a good domain decomposition.

The Oasis3 MPI1/2 Parallel Version

This work describes the optimization and parallelization activities performed on the OASIS3 coupler. The test case used for evaluating and profiling the coupler consists on the CMCCMED coupled model developed by the ANS division of the CMCC and currently in production on the NEC SX9 cluster. The experiments highlighted that the most time consuming transformations are the extrapolation of the fields on the masked points (performed in the extrap function) and interpolation (performed in the scriprmp function). The optimization has been mainly focused on reducing the time spent for I/O operations this reduced the coupling time of 27%. The parallelization of the OASIS3 has been a further step for reducing the elapsed time of the whole coupled model. The proposed parallel approach is based on the distribution of the fields among the available processes. Each process is in charge to apply the coupling transformations on the assigned fields. With this approach the number of coupling fields represents an upper bound to the parallelization level. However this approach can be fully combined with the parallelization based on the geographical domain distribution. The work concludes with a qualitative comparison of the proposed approach with the OASIS3 pseudoparallel version developed by CERFACS.

Operational Wind Wave Prediction System at KMA

An overview of the current operational wind wave prediction system and of new developments at the Korea Meteorological Administration (KMA) is presented. KMA has operated a numerical ocean wave prediction system since 1992. The first major upgrade was done in 1999 with the adaptation of a third-generation wave model (WAM) for both regional (ReWAM-0.25 deg resolution, covering Northeast Asia) and global (GoWAM-1.25 deg resolution) domains. In 2005, the KMA replaced its NEC SX5 computer with a 1024-CPU Cray X1E system, a Parallel Vector Processor (PVP) machine with 128 node modules. A coastal ocean wave prediction system (CoWAM) has been designed and is currently under test mode. Six CoWAM domains, each of 3 deg longitude by 2 deg latitude in size, with a mesh size of 1 km, are nested inside the regional ocean wave prediction system. The directional wave spectra at the boundaries of the CoWAM are provided from a 1/12° resolution upgraded version of the operational ReWAM. The WAVEWATCH-III code (developed at NOAA) is used for the upgraded ReWAM and new CoWAM system. To ensure the required model performance in the Massively Parallel Processor (MPP) architecture of the new supercomputer, a Message Passing Interface (MPI) has been implemented in the model source code. The sea surface wind and significant wave height are verified routinely on a monthly basis. The global moored buoy data (including the coastal buoys operated by KMA) and remote sensing data from Topex/Poseidon, and Jason (retrieved wave heights) and QuikSCAT (retrieved wind data) are used for verification of the wave prediction system. This validation and a comparison of results from the new ReWAM and CoWAM against buoy data off the coast of Korea are discussed.

Reducing Turn-around Times for Supernova Simulations

Many questions in the field of supernova core collapse still remain unanswered due to the complex multi-faceted nature of the problem. A direct computation of the full neutrino radiation hydrodynamics would require a sustained performance of PetaFlop/s and is therefore unfeasible on today’s supercomputers. The modeling required to reduce the computational effort is accompanied by the ambiguity which physical effects are indispensable. As input parameters also contain a certain amount of uncertainty, parameter studies are necessary. For this reason, supernova simulations still require TFlop/s and a careful mapping of the software onto the given hardware is necessary to assure the maximum performance possible. In this paper, we describe the necessary extensions to the partly existing MPI parallelization of the simulation code PROMETHEUS/VERTEX from the Max-Planck Institute for Astrophysics in Garching. With a complete distributed memory parallelization, turn-around times can be decreased substantially. We show for a 15 solar mass model that an efficient usage of up to 32 nodes NEC SX-8 is possible and therefore turn-around times can be reduced by a factor of nearly seven.

Characteristics of an On-Chip Cache on NEC SX Vector Architecture

Thanks to the highly effective memory bandwidth of the vector systems, they can achieve the high computation efficiency for computation-intensive scientific applications. However, they have been encountering the memory wall problem and the effective memory bandwidth rate has decreased, resulting in the decrease in the bytes per flop rates of recent vector systems from 4 (SX-7 and SX-8) to 2 (SX-8R) and 2.5 (SX-9). The situation is getting worse as many functions units and/or cores will be brought into a single chip, because the pin bandwidth is limited and does not scale. To solve the problem, we propose an on-chip cache, called vector cache, to maintain the effective memory bandwidth rate of future vector supercomputers. The vector cache employs a bypass mechanism between the main memory and register files under software controls. We evaluate the performance of the vector cache on the NEC SX vector processor architecture with bytes per flop rates of 2 B/FLOP and 1 B/FLOP, to clarify the basic characteristics of the vector cache. For the evaluation, we use the NEC SX-7 simulator extended with the vector cache mechanism. Benchmark programs for performance evaluation are two DAXPY-like loops and five leading scientific applications. The results indicate that the vector cache boosts the computational efficiencies of the 2 B/FLOP and 1 B/FLOP systems up to the level of the 4 B/FLOP system. Especially, in the case where cache hit rates exceed 50%, the 2 B/FLOP system can achieve a performance comparable to the 4 B/ FLOP system. The vector cache with the bypass mechanism can provide the data both from the main memory and the cache simultaneously. In addition, from the viewpoints of designing the cache, we investigate the impact of cache associativity on the cache hit rate, and the relationship between cache latency and the performance. The results also suggest that the associativity hardly affects the cache hit rate, and the effects of the cache latency depend on the vector loop length of applications. The cache shorter latency contributes to the performance improvement of the applications with shorter loop lengths, even in the case of the 4 B/FLOP system. In the case of longer loop lengths of 256 or more, the latency can effectively be hidden, and the performance is not sensitive to the cache latency. Finally, we discuss the effects of selective caching using the bypass mechanism and loop unrolling on the vector cache performance for the scientific applications. The selective caching is effective for efficient use of the limited cache capacity. The loop unrolling is also effective for the improvement of performance, resulting in a synergistic effect with caching. However, there are exceptional cases; the loop unrolling worsens the cache hit rate due to an increase in the working space to process the unrolled loops over the cache. In this case, an increase in the cache miss rate cancels the gain obtained by unrolling.

Teraflops Sustained Performance With Real World Applications

This paper provides a comprehensive performance evaluation of the NEC SX-8 system at the High Performance Computing Center Stuttgart which has been in operation since July 2005. It provides a description of the installed hardware together with its performance for some synthetic benchmarks and five real world applications. All the applications achieved sustained Tflop/s performance. Additionally, the measurements presented show the ability of the system to solve not only large problems with a very high performance, but also medium sized problems with high efficiency using a large number of processors.

Optimal broadcast for fully connected processor-node networks

We develop and implement an optimal broadcast algorithm for fully connected processor networks under a bidirectional communication model in which each processor can simultaneously send a message to one processor and receive a message from another, possibly different processor. For any number of processors p the algorithm requires N − 1 + ⌈ log p ⌉ communication rounds to broadcast N blocks of data from a root processor to the remaining processors, meeting the lower bound in the model. For data of size m , assuming that sending and receiving data of size m ′ takes time α + β m ′ , the best running time that can be achieved by the division of m into equal-sized blocks is ( ( ⌈ log p ⌉ − 1 ) α + β m ) 2 . The algorithm uses a regular, circulant graph communication pattern, and degenerates into a binomial tree broadcast when the number of blocks to be broadcast is one. The algorithm is furthermore well suited to fully connected clusters of SMP ( Symmetric Multi-Processor) nodes. The algorithm is implemented as part of an MPI ( Message Passing Interface) library. We demonstrate significant practical bandwidth improvements of up to a factor 1.5 over several other, commonly used broadcast algorithms on both a small SMP cluster and a 72 node NEC SX vector supercomputer.

Scientific Application Performance On Leading Scalar and Vector Supercomputering Platforms

The last decade has witnessed a rapid proliferation of superscalar cache-based microprocessors to build high-end computing (HEC) platforms, primarily because of their generality, scalability, and cost effectiveness. However, the growing gap between sustained and peak performance for full-scale scientific applications on conventional supercomputers has become a major concern in high performance computing, requiring significantly larger systems and application scalability than implied by peak performance in order to achieve desired performance. The latest generation of custom-built parallel vector systems have the potential to address this issue for numerical algorithms with sufficient regularity in their computational structure. In this work we explore applications drawn from four areas: magnetic fusion (GTC), plasma physics (LB-MHD-3D), astrophysics (Cactus), and material science (PARATEC). We compare performance of the vector-based Cray X1, X1E, Earth Simulator, NEC SX-8, with performance of three leading commodity-based super-scalar platforms utilizing the IBM Power3, Intel Itanium2, and AMD Opteron processors. Our work makes several significant contributions: a new data-decomposition scheme for GTC that (for the first time) enables a breakthrough of the teraflop barrier; the introduction of a new three-dimensional lattice Boltzmann magneto-hydrodynamic implementation used to study the onset evolution of plasma turbulence that achieves over 26 Tflop/s on 4800 ES processors; the highest per processor performance (by far) achieved by the full-production version of the Cactus ADM-BSSN; and the largest PARATEC cell size atomistic simulation to date. Overall, results show that the vector architectures attain unprecedented aggregate performance across our application suite, demonstrating the tremendous potential of modern parallel vector systems.

Performance evaluation of supercomputers using HPCC and IMB Benchmarks

The HPC Challenge (HPCC) Benchmark suite and the Intel MPI Benchmark (IMB) are used to compare and evaluate the combined performance of processor, memory subsystem and interconnect fabric of five leading supercomputers—SGI Altix BX2, Cray X1, Cray Opteron Cluster, Dell Xeon Cluster, and NEC SX-8. These five systems use five different networks (SGI NUMALINK4, Cray network, Myrinet, InfiniBand, and NEC IXS). The complete set of HPCC Benchmarks are run on each of these systems. Additionally, we present Intel MPI Benchmarks results to study the performance of 11 MPI communication functions on these systems.

A Dynamical Core with Double Fourier Series: Comparison with the Spherical Harmonics Method

Abstract A dynamical core of a general circulation model with the spectral method using double Fourier series (DFS) as basis functions is presented. The model uses the hydrostatic balance approximation and sigma coordinate system in the vertical direction and includes no topography. The model atmosphere is divided into 25 layers with equal sigma depths. Prognostic equations for the vorticity, divergence, temperature, and logarithmic surface pressure are solved by the DFS spectral-transform method with the Fourier filtering at middle and high latitudes. A semi-implicit time-stepping procedure, which deals with the eigendecomposition and inversion of the 3D Helmholtz equation associated with the gravity wave terms, is incorporated for the gravity wave–related terms. The DFS model is tested in terms of the solution of the 3D Helmholtz equation, balanced initial state, developing baroclinic waves, and short- and long-term Held–Suarez–Williamson simulations for T42, T62, T84, and T106 resolutions. It is found that the DFS model is stable and accurate and produces almost the same results as the spherical harmonics method (SHM). The normalized difference (i.e., L2 norm error) measured from the results of highest-resolution SHM-T106 showed a desirable convergence of the DFS solution with the resolution. The convergence property, however, varies with the test case and prognostic variables. The total mass (or global integrated surface pressure) is conserved to a good approximation in the long-term simulations. Computing on the high-performance computer NEC SX-5 (parallel-vector architecture) indicated that DFS is more efficient than the SHM and the efficiency increases with the resolution, for example, by factors of 2.09 and 7.68 for T212 and T1022, respectively.

On the single processor performance of simple lattice Boltzmann kernels

This report presents a comprehensive survey of the effect of different data layouts on the single processor performance characteristics for the lattice Boltzmann method both for commodity “off-the-shelf” (COTS) architectures and tailored HPC systems, such as vector computers. We cover modern 64-bit processors ranging from IA32 compatible (Intel Xeon/Nocona, AMD Opteron), superscalar RISC (IBM Power4), IA64 (Intel Itanium 2) to classical vector (NEC SX6+) and novel vector (Cray X1) architectures. Combining different data layouts with architecture dependent optimization strategies we demonstrate that the optimal implementation strongly depends on the architecture used. In particular, the correct choice of the data layout could supersede complex cache-blocking techniques in our kernels. Furthermore our results demonstrate that vector systems can outperform COTS architectures by one order of magnitude.

Optimisation of a spline based Eulerian--Lagrangian transport solver

Consider solute transport in saturated porous media. Eulerian-Lagrangian methods provide efficient numerical solutions with minimal numerical dispersion. We discuss the optimisation and parallelisation of a serial, spline based Eulerian-Lagrangian code (ELM2D , Fortran 90) on a 64 bit NEC SX-5 platform to support high-resolution numerical experiments. The aim was to reduce execution times by an order of magnitude whilst maintaining numerical accuracy. Profiling analysis indicated potential inefficiencies in the spline and diffusion subsystems of the code. Vectorisation of these subsystems achieved more than an order of magnitude speed increase. Use of either OpenMP or SX-5 microtasking directives was also effective in further reducing computational expense. Benchmarking optimised solutions against 32 bit serial solutions generated on a different platform indicated good numerical agreement of overall solute distributions and plume spatial moments for strongly heterogeneous problems with O(10 7 ) nodes. More than twenty-fold reduction in SX-5 execution times was achieved through vectorisation and parallelisation, but came at the cost of increasing memory demands. A user configurable striping parameter was introduced to the algorithm to determine the in-core storage, yielding a trade-off between execution speed and resource demand.

Scaling first-principles plane-wave codes to thousands of processors

We present some novel computational methods for scaling up first-principles plane-wave based codes to thousands of processors avoiding communication and latency bottlenecks. This allows our code to scale to more processors and larger systems than previous plane-wave codes that are typically limited in scaling to a few hundred processors. We present performance data for the plane-wave pseudopotential code PARATEC (PARAllel Total Energy Code) on the IBM SP, Cray X1, NEC SX6 Earth Simulator in Japan and the SGI Altix. This data shows how plane-wave codes can be scaled to the teraflop regime even though they require global communications on parallel machines.

Porting the 3D gyrokinetic particle-in-cell code GTC to the NEC SX-6 vector architecture: perspectives and challenges

Several years of optimization on the cache-based super-scalar architecture has made it more difficult to port the current version of the 3D particle-in-cell code GTC to the NEC SX-6 vector architecture. This paper explains the initial work that has been done to port this code to the SX-6 computer and to optimize the most time consuming parts. After a few modifications, single-processor results show a performance increase of 5.2 compared to the IBM SP Power3 processor, and 2.7 compared to the Power4.

On the parallelization of boundary element codes using standard and portable libraries

The present paper introduces the main steps towards the parallelization of existing boundary element codes, using standard and portable libraries for writing shared memory parallel programs, OpenMP and LAPACK. Parallel programming techniques can have a great impact on application performance and OpenMP facilitates these improvements. Since, such procedures are not widespread among BEM practitioners, the authors introduce these techniques into an well-known BEM program, described in detail by Brebbia and Dominguez [Boundary Elements: An Introductory Course. CMP, Southampton, 1992]. The code is herein reviewed and rewritten to achieve high performance on shared memory systems. The step-by-step implementation process provides guidelines to develop efficient parallel BEM codes, applicable to many science and engineering problems. Numerical experiments on a SGI Origin 2000 and a NEC SX-6 show the effectiveness of the proposed approach.