HPC Community Research Articles

From the chair

Fall is a busy season for most people and even more so for the HPC community with the flurry of activity leading up to the SC conference in November. As you'll see in this newsletter, there are plenty of exciting opportunities in Denver during SC including acknowledging our award winners, learning more about professional societies at BoFs, and volunteering at the SIGHPC Booth. We look forward to interacting with the next generation of HPC leaders through our student travel grants, early career travel grants, and the HPC Immersion program.

Building a Diverse and Inclusive HPC Community for Mission-Driven Team Science

Building a Diverse and Inclusive HPC Community for Mission-Driven Team Science

Nextflow in Bioinformatics: Executors Performance Comparison Using Genomics Data

Processing big data is a computationally demanding task which has usually been fulfilled by HPC batch systems. These complex systems pose a challenge to scientists due to their cumbersome nature and changing environment. The scientists often lack deeper informatics understanding and experiment reproducibility is increasingly becoming a hard request on the research validity.A new computational paradigm — containers — are meant to contain all dependencies and persist the state which help reproducibility. They have gained a lot of popularity in the informatics community but HPC community remains skeptical and doubts that container platforms are appropriate for demanding tasks or that such infrastructure can reach significant performance.In this paper, we observe the performance of various infrastructure types (HPC, Kubernetes, local) on a Sarek Nextflow bioinformatics workflow with real life genomics data of various sizes. We analyze obtained workload trace and discuss pros and cons of utilized infrastructures. We also show some approaches perform better in terms of available resources but others are more suitable for diversified workflows. Based on the results, we provide recommendations for life science groups which plan to analyze data in large scale.

Identifying challenges and opportunities of in-memory computing on large HPC systems

With the increasing fidelity and resolution enabled by high-performance computing systems, simulation-based scientific discovery is able to model and understand microscopic physical phenomena at a level that was not possible in the past. A grand challenge that the HPC community facing is how to maintain the large amounts of analysis data generated from simulations. In-memory computing, among others, is recognized to be a viable path forward and has experienced tremendous success in the past decade. Nevertheless, there has been a lack of a complete study and understanding of in-memory computing as a whole on HPC systems. Given the enlarging disparity between compute and HPC storage I/O, it is urgent for the HPC community to assess the state of in-memory computing and understand the challenges and opportunities. This paper presents a comprehensive study of in-memory computing with regard to its software evolution, performance, usability, robustness, and portability. In particular, we conduct an indepth analysis on the evolution of in-memory computing based upon more than 3,000 commits, and use realistic workflows for two scientific workloads, i.e., LAMMPS and Laplace to quantitatively assess state-of-the-art in-memory computing libraries, including DataSpaces, DIMES, Flexpath, Decaf and SENSEI on two leading supercomputers, Titan and Cori. Our studies not only illustrate the performance and scalability, but also reveal the key aspects that are of interest to library developers and users, including usability, robustness, portability, potential design defects, etc.

ExCALIBUR—U.K.’s Preparation for the Arrival of the Next Generation of HPC

The articles in this special section focus on the ExCalibur high performance computing program. In 2019, the UK Government funded the Exascale Computing Algorithms and Infrastructures Benefitting UK Research (ExCALIBUR) Programa with an investment of £45.7m, led by the Met Office and EPSRC along with other U.K. research bodies. The vision of ExCALIBUR is to prepare the U.K. hardware and software landscape for the emerging age of exascale computing. The ExCALIBUR priorities are as follows: 1) Separation of Concerns, 2) Co-Design, 3) Data Science, and 4) Investing in People. In 2020, in the first round of ExCALIBUR, 10 software Design and Development Working Groups (DDWG) were formed, each representing a different community and tasked with developing high-priority use cases, the cost-effective strategy to enable these, and demonstrate collaborative progress toward the successful transition to exascale. Significant funds have been invested in 2021 to support a number of these DDWGs for a further three years with some of the DDWGs combining their efforts after identifying common possible solutions to remove underlying bottlenecks in their different applications. These longer three-year projects will see substantial software and other outputs, benefiting the international HPC community. In terms of hardware, the U.K

A fast vectorized sorting implementation based on the ARM scalable vector extension (SVE).

The way developers implement their algorithms and how these implementations behave on modern CPUs are governed by the design and organization of these. The vectorization units (SIMD) are among the few CPUs’ parts that can and must be explicitly controlled. In the HPC community, the x86 CPUs and their vectorization instruction sets were de-facto the standard for decades. Each new release of an instruction set was usually a doubling of the vector length coupled with new operations. Each generation was pushing for adapting and improving previous implementations. The release of the ARM scalable vector extension (SVE) changed things radically for several reasons. First, we expect ARM processors to equip many supercomputers in the next years. Second, SVE’s interface is different in several aspects from the x86 extensions as it provides different instructions, uses a predicate to control most operations, and has a vector size that is only known at execution time. Therefore, using SVE opens new challenges on how to adapt algorithms including the ones that are already well-optimized on x86. In this paper, we port a hybrid sort based on the well-known Quicksort and Bitonic-sort algorithms. We use a Bitonic sort to process small partitions/arrays and a vectorized partitioning implementation to divide the partitions. We explain how we use the predicates and how we manage the non-static vector size. We also explain how we efficiently implement the sorting kernels. Our approach only needs an array of O(log N) for the recursive calls in the partitioning phase, both in the sequential and in the parallel case. We test the performance of our approach on a modern ARMv8.2 (A64FX) CPU and assess the different layers of our implementation by sorting/partitioning integers, double floating-point numbers, and key/value pairs of integers. Our results show that our approach is faster than the GNU C++ sort algorithm by a speedup factor of 4 on average.

Reproducibility Practice in High-Performance Computing: Community Survey Results

The integrity of science and engineering research is grounded in assumptions of rigor and transparency on the part of those engaging in such research. HPC community effort to strengthen rigor and transparency take the form of reproducibility efforts. In a recent survey of the SC conference community, we collected information about the SC reproducibility initiative activities. We present the survey results in this article. Results show that the reproducibility initiative activities have contributed to higher levels of awareness on the part of SC conference technical program participants, and hint at contributing to greater scientific impact for the published papers of the SC conference series. Stringent point-of-manuscript-submission verification is problematic for reasons we point out, as are inherent difficulties of computational reproducibility in HPC. Future efforts should better decouple the community educational goals from goals that specifically strengthen a research work’s potential for long-term impact through reuse 5–10 years down the road.

Beyond MPI

Networkswith Remote DirectMemoryAccess (RDMA) support are becoming increasingly common. RDMA, however, offers a limited programming interface to remote memory that consists of read, write and atomic operations. With RDMA alone, completing the most basic operations on remote data structures often requires multiple round-trips over the network. Data-intensive systems strongly desire higher-level communication abstractions that supportmore complex interaction patterns. A natural candidate to consider is MPI, the de facto standard for developing high-performance applications in the HPC community. This paper critically evaluates the communication primitives of MPI and shows that using MPI in the context of a data processing system comes with its own set of insurmountable challenges. Based on this analysis, we propose a new communication abstraction named RDMO, or Remote DirectMemory Operation, that dispatches a short sequence of reads, writes and atomic operations to remote memory and executes them in a single round-trip.

HeSRPT

Modern data centers serve workloads which can exploit parallelism. When a job parallelizes across multiple servers it completes more quickly. However, it is unclear how to share a limited number of servers between many parallelizable jobs. In this paper we consider a typical scenario where a data center composed of N servers will be tasked with completing a set of M parallelizable jobs. Typically, M is much smaller than N. In our scenario, each job consists of some amount of inherent work which we refer to as a job's size. We assume that job sizes are known up front to the system, and each job can utilize any number of servers at any moment in time. These assumptions are reasonable for many parallelizable workloads such as training neural networks using TensorFlow [2]. Our goal in this paper is to allocate servers to jobs so as to minimize the mean slowdown across all jobs, where the slowdown of a job is the job's completion time divided by its running time if given exclusive access to all N servers. Slowdown measures how a job was interfered with by other jobs in the system, and is often the metric of interest in the theoretical parallel scheduling literature (where it is also called stretch), as well as the HPC community (where it is called expansion factor).

Papers we should be reading

This issue's suggested reading summary comes from member Eitan Frachtenberg, a visiting professor of computer science at Reed College. He and colleague Rhody Kaner are studying gender gaps in the HPC community, and sent this summary of their paper, "Representation of Women High-Performance Computing Conferences."

One Year HPC Certification Forum in Retrospective

The ever-changing nature of HPC has always compelled the HPC community to focus a lot of effort into training of new and existing practitioners. Historically, these efforts were tailored around a typical group of users possessing, due to their background, a certain set of programming skills. However, as HPC has become more diverse in terms of hardware, software and the user background, the traditional training approaches became insufficient in addressing training needs of our community. This increasingly complicated HPC landscape makes development and delivery of new training materials challenging. How should we develop training for users, often coming from non-traditionally HPC disciplines, and only interested in learning a particular set of skills? How can we satisfy their training needs if we don't really understand what these are? It's clear that HPC centres struggle to identify and overcome the gaps in users' knowledge, while users struggle to identify skills required to perform their tasks. With the HPC Certification Forum we aim to clearly categorise, define, and examine competencies expected from proficient HPC practitioners. In this article, we report the status and progress this independent body has made during the first year of its existence. The drafted processes and prototypes are expected to mature into a holistic ecosystem beneficial for all stakeholders in HPC education.

European HPC Landscape

This paper provides an overview on the European HPC landscape supported by a survey, designed by the PRACE-5IP project, accessing more than 50 of the most influential stakeholders of HPC in Europe. It focuses at Tier-0 systems on the European level providing high-end computing and data analysis resources. The different actors are presented and their provided services are analyzed in order to identify overlaps and gaps, complementarity and opportunities for collaborations. A new pan-European HPC portal is proposed in order to get all information on one place and facilitate access to the portfolio of services offered by the European HPC communities.

HPC Processors Benchmarking Assessment for Global System Science Applications

The work undertaken in this paper was done in the Centre of Excellence for Global Systems Science (CoeGSS) – an interdisciplinary project funded by the European Commission. CoeGSS project provides a computer-aided decision support in the face of global challenges (e.g. development of energy, water and food supply systems, urbanisation processes and growth of the cities, pandemic control, etc.) and tries to bring together HPC and global systems science. This paper presents a proposition of GSS benchmark which evaluates HPC architectures with respect to GSS applications and seeks for the best HPC system for typical GSS software environments. The outcome of the analysis is defining a benchmark which represents the average GSS environment and its challenges in a good way: spread of smoking habits and development of tobacco industry, development of green cars market and global urbanisation processes. Results of the tests that have been run on a number of recently appeared HPC platforms allow comparing processors’ architectures with respect to different applications using execution times, TDPs3 and TCOs4 as the basic metrics for ranking HPC architectures. Finally, we believe that our analysis of the results conveys a valuable information to the broadened GSS audience which might help to determine the hardware demands for their specific applications, as well as to the HPC community which requires a mature benchmark set reflecting requirements and traits of the GSS applications. Our work can be considered as a step into direction of development of such mature benchmark.

A performance analysis of the first generation of HPC‐optimized Arm processors

SummaryIn this paper, we present performance results from Isambard, the first production supercomputer to be based on Arm CPUs that have been optimized specifically for HPC. Isambard is the first Cray XC50 “Scout” system, combining Cavium ThunderX2 Arm‐based CPUs with Cray's Aries interconnect. The full Isambard system will be delivered in the summer of 2018, when it will contain over 10 000 Arm cores. In this work, we present node‐level performance results from eight early‐access nodes that were upgraded to B0 beta silicon in March 2018. We present node‐level benchmark results comparing ThunderX2 with mainstream CPUs, including Intel Skylake and Broadwell, as well as Xeon Phi. We focus on a range of applications and mini‐apps important to the UK national HPC service, ARCHER, as well as to the Isambard project partners and the wider HPC community. We also compare performance across three major software toolchains available for Arm: Cray's CCE, Arm's version of Clang/Flang/LLVM, and GNU.

Towards an HPC Certification Program

The HPC community has always considered the training of new and existing HPC practitioners to be of high importance to its growth. This diversification of HPC practitioners challenges the traditional training approaches, which are not able to satisfy the specific needs of users, often coming from non-traditionally HPC disciplines, and only interested in learning a particular set of competences. Challenges for HPC centres are to identify and overcome the gaps in users’ knowledge, while users struggle to identify relevant skills. We have developed a first version of an HPC certification program that would clearly categorize, define, and examine competences. Making clear what skills are required of or recommended for a competent HPC user would benefit both the HPC service providers and practitioners. Moreover, it would allow centres to bundle together skills that are most beneficial for specific user roles and scientific domains. From the perspective of content providers, existing training material can be mapped to competences allowing users to quickly identify and learn the skills they require. Finally, the certificates recognized by the whole HPC community simplify inter-comparison of independently offered courses and provide additional incentive for participation.

A Novel DSP Architecture for Scientific Computing and Deep Learning

Exascale computing requires accelerators with ultrahigh power efficiency. Digital signal processors (DSPs), the most important embedded processors widely known for high power efficiency, are rarely explored in the HPC community. We propose a 64-bit general purpose DSP architecture, FT-Matrix2000, which not only integrates the main features of DSPs but also presents several novel enhancements for scientific computing. The FT-Matrix2000 architecture comprises multiple FT-Matrix2 cores and optional RISC CPU cores. The FT-Matrix2 core utilizes a VLIW+SIMD architecture, provides support for double precision operations, and optimizes both the data and control path for scientific computing. Our evaluations show that the performance and efficiency of FT-Matrix2000 are 1107GFLOPS and 92.25%. Compared with the MIC and a 40nm process GPU, FT-Matrix2000 improves the GEMM power efficiency with a factor of 1.49 and 2.68, respectively. We build up a prototype supercomputer with FT-Matrix2000/12. Its HPL efficiency achieves 62.2%, and the performance power ratio is 5.33 GFLOPS/W, which can rank the fourth in the latest Green500 list. These results validate that the FT-Matrix2000 architecture is suitable for scientific computing while maintaining the efficiency of signal processing well. Moreover, the enhancement of FT-Matrix2000 in vector and matrix related computations also enable it to efficiently support deep learning related applications. We have implemented some typical DCNN models on FT-Matrx2000, NVIDIA GPUs, and Vision P6 DSP. The experiments demonstrate that the average computation efficiency of the proposed architecture based on Matrix2000 is about 20 ~ 35% and 8% higher respectively than GPUs and Cadence Vision P6 DSP.

Best practices for management and operation of large HPC installations

SummaryTo achieve their mission and goals, HPC centers continually strive to improve the effectiveness of their resources and services to best serve their constituencies. Collectively, the community has learned a great deal about how to manage and operate HPC centers, provide robust and effective services, and develop new communities as well as about other important aspects. Yet, cataloguing best practices to help inform and guide the broader HPC community is not often done. To improve the situation, the Blue Waters project has documented sets of best practices that have been adopted for the deployment and operation over the past five years of the Blue Waters leadership system, a large Cray XE6/XK7 supercomputer at NCSA. Those practices, described in this paper, cover aspects of managing and operating the system and its resources, supporting its users, and expanding the diversity of applications and communities. Although the technical practices are sometimes discussed relative to Cray systems and leadership‐scale systems, we believe that they would benefit the deployment and operation of other large HPC installations as well.

Autotuning in High-Performance Computing Applications

Autotuning refers to the automatic generation of a search space of possible implementations of a computation that are evaluated through models and/or empirical measurement to identify the most desirable implementation. Autotuning has the potential to dramatically improve the performance portability of petascale and exascale applications. To date, autotuning has been used primarily in high-performance applications through tunable libraries or previously tuned application code that is integrated directly into the application. This paper draws on the authors’ extensive experience applying autotuning to high-performance applications, describing both successes and future challenges. If autotuning is to be widely used in the HPC community, researchers must address the software engineering challenges, manage configuration overheads, and continue to demonstrate significant performance gains and portability across architectures. In particular, tools that configure the application must be integrated into the application build process so that tuning can be reapplied as the application and target architectures evolve.

SC18 selects the next reproducibility challenge paper

The HPC community continues to recognize the critical role that replication and reproducibility play in reesearch. The SC Conference series, through both its Technical Program and Student Cluster Competition (SCC), has been a pioneer in evaluating HPC applications in terms of their replicability.

SIGHPC education brings many gifts to serve HPC community

This year, the SIGHPC Education Chapter received significant gifts: a set of new leaders to take the reins from the pioneers of the SIG Chapter; the joining of memberships between the International HPC Training Consortium and SIGHPC Education, which doubled our numbers; and, finally, the energetic members of SIGHPC who are stepping up and contributing to key focus areas. We'd like to share these gifts with all of you.