Petascale Computing Research Articles

HFBN: An Energy Efficient High Performance Hierarchical Interconnection Network for Exascale Supercomputer

Supercomputers are trying to be eco-friendly using their main components like- interconnection networks, processors, and shared memory through high power efficiency (GFlops/watts). The reason for this is the exascale systems, are on the horizon, require a 1000 times performance improvement over the petascale computers and energy efficiency has attracted as the key factor for to achieve exascale system. The main contribution of this paper is to introduce a new hierarchical interconnection network; considering simulations at the million processing cores especially for the exascale system. Moreover, our designed network claims the supremacy of high network performance and low power consumption over the conventional networks and ensuring the utmost preferability for exaFLOPS system. On the other hand, the performance per watt metric, used for the TOP500 list, doesn’t reflect the overall performance of a given system. Hence, one of the possible solutions to reach the next generation exascale performance is to redesign the "Interconnection Network", which holds the main responsibility for the intercommunication between the CPUs and also the power consumption for the supercomputers. This paper focuses on a redesigned new energy efficient interconnection network that mitigates the problems of high power consumption, longer wiring length, and low bandwidth issues. Our designed network (HFBN) has been compared against the Tofu network and in the case of 1M cores, HFBN can obtain about 87.26% better energy efficiency at the zero load latency with uniform traffic, about 86.32% with perfect shuffle traffic, and about 92.98% with the bit-compliment traffic.

Scalable reservoir computing on coherent linear photonic processor

Photonic neuromorphic computing is of particular interest due to its significant potential for ultrahigh computing speed and energy efficiency. The advantage of photonic computing hardware lies in its ultrawide bandwidth and parallel processing utilizing inherent parallelism. Here, we demonstrate a scalable on-chip photonic implementation of a simplified recurrent neural network, called a reservoir computer, using an integrated coherent linear photonic processor. In contrast to previous approaches, both the input and recurrent weights are encoded in the spatiotemporal domain by photonic linear processing, which enables scalable and ultrafast computing beyond the input electrical bandwidth. As the device can process multiple wavelength inputs over the telecom C-band simultaneously, we can use ultrawide optical bandwidth (~5 terahertz) as a computational resource. Experiments for the standard benchmarks showed good performance for chaotic time-series forecasting and image classification. The device is considered to be able to perform 21.12 tera multiplication–accumulation operations per second (MAC ∙ s−1) for each wavelength and can reach petascale computation speed on a single photonic chip by using wavelength division multiplexing. Our results are challenging for conventional Turing–von Neumann machines, and they confirm the great potential of photonic neuromorphic processing towards peta-scale neuromorphic super-computing on a photonic chip.

Wall resolved large eddy simulation of reactor core flows with the spectral element method

Numerical simulation is an intrinsic part of nuclear engineering research supporting the design of nuclear power plants. With the advent of Petascale computing (i.e., computers capable of more than 1 PFlop), the simulation of portions of reactor components with turbulence-resolving techniques is now possible. These simulations can provide invaluable insight into the flow dynamics, which is difficult or often impossible to obtain with experiments alone. The spectral element method in particular has emerged as a powerful method to deliver massively parallel calculations at high fidelity by using Large Eddy Simulation or Direct Numerical Simulation. In this work we review the fundamentals of the method and the reasons it is compelling for the simulation of nuclear reactor core flows. We review a series of wall-resolved large eddy simulations of reactor cores applied to three reactor types. In particular we discuss the simulation of the flow in a 17 × 17 rod bundle related to light water reactor (LWR) applications; the flow in a 37-pin wire-wrapped rod bundle related to liquid metal reactor (LMR) applications and the flow through a graphite block with application to molten salt reactors (MSR).

FFT, FMM, and multigrid on the road to exascale: Performance challenges and opportunities

FFT, FMM, and multigrid methods are widely used fast and highly scalable solvers for elliptic PDEs. However, emerging large-scale computing systems are introducing challenges in comparison to current petascale computers. Recent efforts (Dongarra et al. 2011) have identified several constraints in the design of exascale software that include massive concurrency, resilience management, exploiting the high performance of heterogeneous systems, energy efficiency, and utilizing the deeper and more complex memory hierarchy expected at exascale. In this paper, we perform a model-based comparison of the FFT, FMM, and multigrid methods in the context of these projected constraints. In addition we use performance models to offer predictions about the expected performance on upcoming exascale system configurations based on current technology trends.

Wormhole optical network: a new architecture to solve long diameter problem in exascale computer

The exascale computer will be built in the near future thanks to rapid innovations in semiconductor logic, memory, architectures, interconnections and other essential technologies. It is difficult to design an interconnection network that combines high performance with low power consumption. Therefore, building an interconnection network with high cost performance plays a critical role in building such a large scale system. Currently, torus-interconnected network like 6D-Torus possesses suitable properties for the petascale computer. However, the diameter within the torus-interconnected network is too long to achieve efficient global communication in the exascale computer. In addition, a direct connection method is not adaptive to the diverse characteristics of traffic. Here, we propose an architecture called Wormhole Optical Network (WON) for the exascale computer which is based on optical circuit switching. WON was designed to fully integrate into the electrical network of 6D-Torus. WON allows for the use of three novel technologies, namely the dynamic topology with optical links, algorithm of cross dimension order routing, and strategy of flow control for deadlock-free. We evaluated WON using both a prototype system and a simulator for the exascale computer. Our analysis shows that compared to the traditional electrical architecture, WON architecture reduced the time of data communication by 14–29% on exascale, a result obtained for a wide selection of diverse applications. Through enabling an SDN controller to adjust topology, WON maintains high utilization of optical links for inter-process communication from diverse applications. Further, we quantified WON’s flexibility of job deployment for mitigating hotspot traffic. We show that WON reduced latency by 20–35% in the large-range deployment and improved throughput by 30% in the long-distance deployment.

A Cognitive Framework to Secure Smart Cities

The advancement in technology has transformed Cyber Physical Systems and their interface with IoT into a more sophisticated and challenging paradigm. As a result, vulnerabilities and potential attacks manifest themselves considerably more than before, forcing researchers to rethink the conventional strategies that are currently in place to secure such physical systems. This manuscript studies the complex interweaving of sensor networks and physical systems and suggests a foundational innovation in the field. In sharp contrast with the existing IDS and IPS solutions, in this paper, a preventive and proactive method is employed to stay ahead of attacks by constantly monitoring network data patterns and identifying threats that are imminent. Here, by capitalizing on the significant progress in processing power (e.g. petascale computing) and storage capacity of computer systems, we propose a deep learning approach to predict and identify various security breaches that are about to occur. The learning process takes place by collecting a large number of files of different types and running tests on them to classify them as benign or malicious. The prediction model obtained as such can then be used to identify attacks. Our project articulates a new framework for interactions between physical systems and sensor networks, where malicious packets are repeatedly learned over time while the system continually operates with respect to imperfect security mechanisms.

Parcels v0.9: prototyping a Lagrangian ocean analysis framework for the petascale age

Abstract. As ocean general circulation models (OGCMs) move into the petascale age, where the output of single simulations exceeds petabytes of storage space, tools to analyse the output of these models will need to scale up too. Lagrangian ocean analysis, where virtual particles are tracked through hydrodynamic fields, is an increasingly popular way to analyse OGCM output, by mapping pathways and connectivity of biotic and abiotic particulates. However, the current software stack of Lagrangian ocean analysis codes is not dynamic enough to cope with the increasing complexity, scale and need for customization of use-cases. Furthermore, most community codes are developed for stand-alone use, making it a nontrivial task to integrate virtual particles at runtime of the OGCM. Here, we introduce the new Parcels code, which was designed from the ground up to be sufficiently scalable to cope with petascale computing. We highlight its API design that combines flexibility and customization with the ability to optimize for HPC workflows, following the paradigm of domain-specific languages. Parcels is primarily written in Python, utilizing the wide range of tools available in the scientific Python ecosystem, while generating low-level C code and using just-in-time compilation for performance-critical computation. We show a worked-out example of its API, and validate the accuracy of the code against seven idealized test cases. This version 0.9 of Parcels is focused on laying out the API, with future work concentrating on support for curvilinear grids, optimization, efficiency and at-runtime coupling with OGCMs.

Petascale supercomputing to accelerate the design of high-temperature alloys

Recent progress in high-performance computing and data informatics has opened up numerous opportunities to aid the design of advanced materials. Herein, we demonstrate a computational workflow that includes rapid population of high-fidelity materials datasets via petascale computing and subsequent analyses with modern data science techniques. We use a first-principles approach based on density functional theory to derive the segregation energies of 34 microalloying elements at the coherent and semi-coherent interfaces between the aluminium matrix and the θ′-Al2Cu precipitate, which requires several hundred supercell calculations. We also perform extensive correlation analyses to identify materials descriptors that affect the segregation behaviour of solutes at the interfaces. Finally, we show an example of leveraging machine learning techniques to predict segregation energies without performing computationally expensive physics-based simulations. The approach demonstrated in the present work can be applied to any high-temperature alloy system for which key materials data can be obtained using high-performance computing.

Constructing Neuronal Network Models in Massively Parallel Environments.

Recent advances in the development of data structures to represent spiking neuron network models enable us to exploit the complete memory of petascale computers for a single brain-scale network simulation. In this work, we investigate how well we can exploit the computing power of such supercomputers for the creation of neuronal networks. Using an established benchmark, we divide the runtime of simulation code into the phase of network construction and the phase during which the dynamical state is advanced in time. We find that on multi-core compute nodes network creation scales well with process-parallel code but exhibits a prohibitively large memory consumption. Thread-parallel network creation, in contrast, exhibits speedup only up to a small number of threads but has little overhead in terms of memory. We further observe that the algorithms creating instances of model neurons and their connections scale well for networks of ten thousand neurons, but do not show the same speedup for networks of millions of neurons. Our work uncovers that the lack of scaling of thread-parallel network creation is due to inadequate memory allocation strategies and demonstrates that thread-optimized memory allocators recover excellent scaling. An analysis of the loop order used for network construction reveals that more complex tests on the locality of operations significantly improve scaling and reduce runtime by allowing construction algorithms to step through large networks more efficiently than in existing code. The combination of these techniques increases performance by an order of magnitude and harnesses the increasingly parallel compute power of the compute nodes in high-performance clusters and supercomputers.

Performance analysis of open‐source distributed file systems for practical large‐scale molecular ab initio, density functional theory, and GW + BSE calculations

AbstractApplication of conventional post‐Hartree–Fock theory, density functional theory, and GW + BSE theory, on chemical and biological problems of growing molecular scale, has created a need for innovative designs in high‐performance computing infrastructures and associated middleware technologies. The growing trend toward petascale computing creates an enormous demand for highly efficient storage solutions. Notably, in reaching limitations of the scale‐up model, technological progress, and continual growth in demand has evolved into solutions that alternatively scale‐out. This investigation involves an analysis of commonly used open source storage solutions for high demand ab initio calculations. Comparisons are shown for the most competitive solutions for a broad range of workloads, following disk usage of key ab initio algorithms in the widely used open source electronic structure theory software, GAMESS. Assessment is based on results of read/write speeds and ratios, scaling ability, and impact of block‐size on overall performance of the distributed storage. GlusterFS is shown to generally outperform CEPH and local disk file systems, providing the most reasonable alternative to local storage, and is therefore recommended for large‐scale molecular calculations.

Methodology for the Simulation of Molecular Motors at Different Scales.

Millisecond-scale conformational transitions represent a seminal challenge for traditional molecular dynamics simulations, even with the help of high-end supercomputer architectures. Such events are particularly relevant to the study of molecular motors-proteins or abiological constructs that convert chemical energy into mechanical work. Here, we present a hybrid-simulation scheme combining an array of methods including elastic network models, transition path sampling, and advanced free-energy methods, possibly in conjunction with generalized-ensemble schemes to deliver a viable option for capturing the millisecond-scale motor steps of biological motors. The methodology is already implemented in large measure in popular molecular dynamics programs, and it can leverage the massively parallel capabilities of petascale computers. The applicability of the hybrid method is demonstrated with two examples, namely cyclodextrin-based motors and V-type ATPases.

Large-scale large eddy simulation of nuclear reactor flows: Issues and perspectives

Numerical simulation has been an intrinsic part of nuclear engineering research since its inception. In recent years a transition is occurring toward predictive, first-principle-based tools such as computational fluid dynamics. Even with the advent of petascale computing, however, such tools still have significant limitations. In the present work some of these issues, and in particular the presence of massive multiscale separation, are discussed, as well as some of the research conducted to mitigate them.Petascale simulations at high fidelity (large eddy simulation/direct numerical simulation) were conducted with the massively parallel spectral element code Nek5000 on a series of representative problems. These simulations shed light on the requirements of several types of simulation: (1) axial flow around fuel rods, with particular attention to wall effects; (2) natural convection in the primary vessel; and (3) flow in a rod bundle in the presence of spacing devices. The focus of the work presented here is on the lessons learned and the requirements to perform these simulations at exascale. Additional physical insight gained from these simulations is also emphasized.

Computational sciences in the upstream oil and gas industry.

The predominant technical challenge of the upstream oil and gas industry has always been the fundamental uncertainty of the subsurface from which it produces hydrocarbon fluids. The subsurface can be detected remotely by, for example, seismic waves, or it can be penetrated and studied in the extremely limited vicinity of wells. Inevitably, a great deal of uncertainty remains. Computational sciences have been a key avenue to reduce and manage this uncertainty. In this review, we discuss at a relatively non-technical level the current state of three applications of computational sciences in the industry. The first of these is seismic imaging, which is currently being revolutionized by the emergence of full wavefield inversion, enabled by algorithmic advances and petascale computing. The second is reservoir simulation, also being advanced through the use of modern highly parallel computing architectures. Finally, we comment on the role of data analytics in the upstream industry.This article is part of the themed issue 'Energy and the subsurface'.

Atomic detail visualization of photosynthetic membranes with GPU-accelerated ray tracing

The cellular process responsible for providing energy for most life on Earth, namely photosynthetic light-harvesting, requires the cooperation of hundreds of proteins across an organelle, involving length and time scales spanning several orders of magnitude over quantum and classical regimes. Simulation and visualization of this fundamental energy conversion process pose many unique methodological and computational challenges. We present, in two accompanying movies, light-harvesting in the photosynthetic apparatus found in purple bacteria, the so-called chromatophore. The movies are the culmination of three decades of modeling efforts, featuring the collaboration of theoretical, experimental, and computational scientists. We describe the techniques that were used to build, simulate, analyze, and visualize the structures shown in the movies, and we highlight cases where scientific needs spurred the development of new parallel algorithms that efficiently harness GPU accelerators and petascale computers.

Big Data: Next-Generation Machines for Big Science

Addressing the scientific grand challenges identified by the US Department of Energy's (DOE's) Office of Science's programs alone demands a total leadership-class computing capability of 150 to 400 Pflops by the end of this decade. The successors to three of the DOE's most powerful leadership-class machines are set to arrive in 2017 and 2018-the products of the Collaboration Oak Ridge Argonne Livermore (CORAL) initiative, a national laboratory-industry design/build approach to engineering next-generation petascale computers for grand challenge science. These mission-critical machines will enable discoveries in key scientific fields such as energy, biotechnology, nanotechnology, materials science, and high-performance computing, and serve as a milestone on the path to deploying exascale computing capabilities.

Direct numerical simulations in solid mechanics for understanding the macroscale effects of microscale material variability

Abstract A fundamental challenge for the quantification of uncertainty in solid mechanics is understanding how microscale material variability is manifested at the macroscale. In an era of petascale computing and future exascale computing, it is now possible to perform direct numerical simulations (DNS) in solid mechanics where the microstructure is modeled directly in a macroscale structure. Using this DNS capability, we investigate the macroscale response of polycrystalline microstructures and the accuracy of homogenization theory for upscaling the microscale response. Using a massively parallel finite-element code, we perform an ensemble of direct numerical simulations in which polycrystalline microstructures are embedded throughout a macroscale structure. The largest simulations model approximately 420 thousand grains within an I-beam. The inherently random DNS results are compared with corresponding simulations based on the deterministic governing equations and material properties obtained from homogenization theory. Evidence is sought for both surface effects and other higher-order effects as predicted by homogenization theory for macroscale structures containing finite microstructures.

A Scalable $O(N)$ Algorithm for Large-Scale Parallel First-Principles Molecular Dynamics Simulations

Traditional algorithms for first-principles molecular dynamics (FPMD) simulations only gain a modest capability increase from current petascale computers, due to their $O(N^3)$ complexity and their heavy use of global communications. To address this issue, we are developing a truly scalable $O(N)$ complexity FPMD algorithm, based on density functional theory (DFT), which avoids global communications. The computational model uses a general nonorthogonal orbital formulation for the DFT energy functional, which requires knowledge of selected elements of the inverse of the associated overlap matrix. We present a scalable algorithm for approximately computing selected entries of the inverse of the overlap matrix, based on an approximate inverse technique, by inverting local blocks corresponding to principal submatrices of the global overlap matrix. The new FPMD algorithm exploits sparsity and uses nearest neighbor communication to provide a computational scheme capable of extreme scalability. Accuracy is contr...

Huge-scale molecular dynamics simulation of multibubble nuclei

We have developed molecular dynamics codes for a short-range interaction potential that adopt both the flat-MPI and MPI/OpenMP hybrid parallelizations on the basis of a full domain decomposition strategy. Benchmark simulations involving up to 38.4 billion Lennard-Jones particles were performed on Fujitsu PRIMEHPC FX10, consisting of 4800 SPARC64 IXfx 1.848 GHz processors, at the Information Technology Center of the University of Tokyo, and a performance of 193 teraflops was achieved, which corresponds to a 17.0% execution efficiency. Cavitation processes were also simulated on PRIMEHPC FX10 and SGI Altix ICE 8400EX at the Institute of Solid State Physics of the University of Tokyo, which involved 1.45 billion and 22.9 million particles, respectively. Ostwald-like ripening was observed after the multibubble nuclei. Our results demonstrate that direct simulations of multiscale phenomena involving phase transitions from the atomic scale are possible and that the molecular dynamics method is a promising method that can be applied to petascale computers.

Analysis and parallel implementation of a forced N-body problem

The understanding of particle dynamics in N-body problems is of importance to many applications in astrophysics, molecular dynamics and cloud/plasma physics where the theoretical representation results in a coupled system of equations for a large number of entities. This paper concerns algorithms for solving a specific N-body problem, namely, a system of disturbance velocities for hydrodynamically interacting particles in a particle-laden turbulent flow. The system is derived from the improved superposition method of [1]. Targeting for scalable computations on petascale computers, we have carried out a thorough study of a parallel implementation of GMRes with different features, such as preconditioners, matrix-free and parallel sparse representation of the matrix through 1D and 2D spatial domain decompositions. Gauss–Seidel method is also studied as a reference iterative algorithm. The range of conditions for efficiency and failure of each method is discussed in detail.Through perturbation analysis, we have conducted a series of experiments to understand the effect of particle sizes, interaction symmetry, inter-particle distances and interaction truncation on the eigenvalues and normality of the linear system. For situations where the system is ill-conditioned, we introduce a restricted Schwarz type preconditioner. We verified the parallel efficiency of the preconditioner using 1D domain decomposition on a parallel machine. A benchmark problem of particle laden turbulence at 5123 resolution with 2×106 particles is studied to understand the scalability of the proposed methods on parallel machines. We have developed a stable and highly scalable parallel solver with an affordable computational cost even for ill-conditioned systems through preconditioning. On 64 cores, using GMRes in 2D domain decomposition, we achieved a speed-up of ∼5.6x (relative to 1D domain decomposition on the same number of processors). Our complexity analysis showed that for large N-body problems, the proposed GMRes scheme scales well for moderate to large number of processors in current tera to petascale computers.

Towards petascale spectral simulations for transition analysis in wall bounded flow

SUMMARYAn efficient parallel spectral method for direct numerical simulations of transitional and turbulent flows is described in this paper. The parallelization is classically based on a bidimensional domain decomposition, but has been specifically developed for a solenoidal Fourier–Chebyshev spectral approximation where in one Fourier direction, the number of modes is very large compared with the two other directions. The approach therefore differs from classical libraries developed for cubic Fourier boxes. The strategy uses message‐passing interface (MPI) for message‐passing among nodes and is fairly portable. One of the originalities of this paper is the use of an efficient hybrid programming with MPI for internodes communications and a coarse grain parallelism using OpenMP for core shared‐memory computation, instead of the classical hybrid programming with MPI and a fine granularity parallelism at the loop level with OpenMP directives. This hybrid parallelism has been tested on the recent generation of high‐performance parallel supercomputers involving a few tens of cores per node. Performances are evaluated on different low‐frequency and high‐frequency processors massively parallel platforms. We demonstrate that spectral methods, which are known to be inherently ill‐fitted for the new generation of high‐performance distributed‐memory computers, can be implemented efficiently using this hybrid programming with good scalability and a very fast wall‐clock time per iteration. New numerical experiments are therefore now accessible on petascale computers, while keeping the attractive features of spectral methods such as accuracy, exponential convergence, computational efficiency and conservative properties. This is illustrated by a direct numerical simulation of the transition of the boundary layers developing from the entrance section of a plane channel and interacting to merge into a fully turbulent flow. Copyright © 2012 John Wiley & Sons, Ltd.