Coarray Fortran Research Articles

Accelerating Fortran codes: A method for integrating Coarray Fortran with CUDA Fortran and OpenMP

Fortran's prominence in scientific computing requires strategies to ensure both that legacy codes are efficient on high-performance computing systems, and that the language remains attractive for the development of new high-performance codes. Coarray Fortran (CAF), part of the Fortran 2008 standard introduced for parallel programming, facilitates distributed memory parallelism with a syntax familiar to Fortran programmers, simplifying the transition from single-processor to multi-processor coding. This research focuses on innovating and refining a parallel programming methodology that fuses the strengths of Intel Coarray Fortran, Nvidia CUDA Fortran, and OpenMP for distributed memory parallelism, high-speed GPU acceleration and shared memory parallelism respectively. We consider the management of pageable and pinned memory, CPU-GPU affinity in NUMA multiprocessors, and robust compiler interfacing with speed optimisation. We demonstrate our method through its application to a parallelised Poisson solver and compare the methodology, implementation, and scaling performance to that of the Message Passing Interface (MPI), finding CAF offers similar speeds with easier implementation. For new codes, this approach offers a faster route to optimised parallel computing. For legacy codes, it eases the transition to parallel computing, allowing their transformation into scalable, high-performance computing applications without the need for extensive re-design or additional syntax.

Parallel SnowModel (v1.0): a parallel implementation of a distributed snow-evolution modeling system (SnowModel)

Abstract. SnowModel, a spatially distributed snow-evolution modeling system, was parallelized using Coarray Fortran for high-performance computing architectures to allow high-resolution (1 m to hundreds of meters) simulations over large regional- to continental-scale domains. In the parallel algorithm, the model domain was split into smaller rectangular sub-domains that are distributed over multiple processor cores using one-dimensional decomposition. All the memory allocations from the original code were reduced to the size of the local sub-domains, allowing each core to perform fewer computations and requiring less memory for each process. Most of the subroutines in SnowModel were simple to parallelize; however, there were certain physical processes, including blowing snow redistribution and components within the solar radiation and wind models, that required non-trivial parallelization using halo-exchange patterns. To validate the parallel algorithm and assess parallel scaling characteristics, high-resolution (100 m grid) simulations were performed over several western United States domains and over the contiguous United States (CONUS) for a year. The CONUS scaling experiment had approximately 70 % parallel efficiency; runtime decreased by a factor of 1.9 running on 1800 cores relative to 648 cores (the minimum number of cores that could be used to run such a large domain because of memory and time limitations). CONUS 100 m simulations were performed for 21 years (2000–2021) using 46 238 and 28 260 grid cells in the x and y dimensions, respectively. Each year was simulated using 1800 cores and took approximately 5 h to run.

Improvement in Runtime Speed for Frequency Domain Soil–Structure Interaction Analysis Using a Coarray Parallel Processing Technique

In this paper, we propose a new algorithm for introducing Coarray Fortran (CAF) into a program that dynamically analyzes the frequency domain in soil–structure interactions. This was implemented in KIESSI-3D, which is frequency domain soil-structure interaction analysis computer code based on finite and infinite element techniques, and the performance of analysis speed based on the number of images (or coarrays) and cores per image was evaluated. For the performance evaluation, we used examples of full-scale nuclear power plant buildings with a shallow foundation model as well as a deep foundation model. In terms of analysis speed, the new KIESSI-3D improved the analysis speed by an average of 2.78 times for the shallow foundation problem and 2.69 times for the deep foundation problem, compared with the existing KIESSI-3D, because of the new algorithm and CAF. Further, as the number of cores and the internal memory size in the computer systems increased, the efficiency of also parallel processing increased.

A New Parallel Code Based on a Simple Piecewise Parabolic Method for Numerical Modeling of Colliding Flows in Relativistic Hydrodynamics

A new parallel code based on models of special relativistic hydrodynamics is presented for describing interacting flows. A new highly accurate numerical method is considered and verified. A parallel implementation of the method by means of Coarray Fortran technology and its efficiency are described in detail. The code scalability is 92% on a cluster with Intel Xeon 6248R NKS-1P with 192 Coarray Fortran images. Different interacting relativistic flows are considered as astrophysical applications.

Parallel Hybrid Simulations of Block Copolymer Nanocomposites using Coarray Fortran

Back Cover: A parallel cell dynamic simulation algorithm is presented for hybrid block copolymer/nanoparticle systems, using Coarray Fortran. It allows to scale up to hundreds of processors, while displaying no drawbacks against an MPI implementation. As a result, previously unavailable system sizes can be studied which is crucial to analyse the block copolymer nanocomposite structure. This is reported by Ignacio Pagonabarraga and co-workers in article number 2100007.

Parallel Hybrid Simulations of Block Copolymer Nanocomposites using Coarray Fortran

AbstractComputer simulations of experimentally comparable system sizes in soft matter often require considerable elapsed times. The use of many cores can reduce the needed time, ideally proportionally to the number of processors. In this paper a parallel computational method using coarray Fortran is implemented and tested for large systems of purely block copolymer melts, as well as block copolymer nanocomposites. A satisfactory strong scaling is shown up to 512 cores while a weak scaling with a drop in performance is achieved up to 4096 cores. The scaling of the parallel cell dynamic simulations scheme displays no drawbacks over MPI and provides an example of the simplicity of the coarray approach. The code has been tested on several architectures and compilers. The hybrid block copolymer/nanoparticle algorithm can achieve previously unavailable system sizes.

A comparison of MPI and co-array FORTRAN for large finite element variably saturated flow simulations

The purpose of this research is to determine how well co-array FORTRAN (CAF) performs relative to Message Passing Interface (MPI) on unstructured mesh finite element groundwater modelling applications with large problem sizes and core counts. This research used almost 150 million nodes and 300 million 3-D prism elements. Results for both the Cray XE6 and Cray XC30 are given. A comparison of the ghost-node update algorithms with source code provided for both MPI and CAF is also presented.

A highly scalable particle tracking algorithm using partitioned global address space (PGAS) programming for extreme-scale turbulence simulations

A new parallel algorithm utilizing a partitioned global address space (PGAS) programming model to achieve high scalability is reported for particle tracking in direct numerical simulations of turbulent fluid flow. The work is motivated by the desire to obtain Lagrangian information necessary for the study of turbulent dispersion at the largest problem sizes feasible on current and next-generation multi-petaflop supercomputers. A large population of fluid particles is distributed among parallel processes dynamically, based on instantaneous particle positions such that all of the interpolation information needed for each particle is available either locally on its host process or neighboring processes holding adjacent sub-domains of the velocity field. With cubic splines as the preferred interpolation method, the new algorithm is designed to minimize the need for communication, by transferring between adjacent processes only those spline coefficients determined to be necessary for specific particles. This transfer is implemented very efficiently as a one-sided communication, using Co-Array Fortran (CAF) features which facilitate small data movements between different local partitions of a large global array. The cost of monitoring transfer of particle properties between adjacent processes for particles migrating across sub-domain boundaries is found to be small. Detailed benchmarks are obtained on the Cray petascale supercomputer Blue Waters at the University of Illinois, Urbana–Champaign. For operations on the particles in a 81923 simulation (0.55 trillion grid points) on 262,144 Cray XE6 cores, the new algorithm is found to be orders of magnitude faster relative to a prior algorithm in which each particle is tracked by the same parallel process at all times. This large speedup reduces the additional cost of tracking of order 300 million particles to just over 50% of the cost of computing the Eulerian velocity field at this scale. Improving support of PGAS models on major compilers suggests that this algorithm will be of wider applicability on most upcoming supercomputers.

Coarray-based load balancing on heterogeneous and many-core architectures

Abstract In order to reach challenging performance goals, computer architecture is expected to change significantly in the near future. Heterogeneous chips, equipped with different types of cores and memory, will force application developers to deal with irregular communication patterns, high levels of parallelism, and unexpected behavior. Load balancing among the heterogeneous compute units will be a critical task in order to achieve an effective usage of the computational power provided by such new architectures. In this highly dynamic scenario, Partitioned Global Address Space (PGAS) languages, like Coarray Fortran, appear a promising alternative to standard MPI programming that uses two-sided communications, in particular because of PGAS one-sided semantic and ease of programmability. In this paper, we show how Coarray Fortran can be used for implementing dynamic load balancing algorithms on an exascale compute node and how these algorithms can produce performance benefits for an Asian option pricing problem, running in symmetric mode on Intel Xeon Phi Knights Corner and Knights Landing architectures.

MPI to Coarray Fortran: Experiences with a CFD Solver for Unstructured Meshes

High-resolution numerical methods and unstructured meshes are required in many applications of Computational Fluid Dynamics (CFD). These methods are quite computationally expensive and hence benefit from being parallelized. Message Passing Interface (MPI) has been utilized traditionally as a parallelization strategy. However, the inherent complexity of MPI contributes further to the existing complexity of the CFD scientific codes. The Partitioned Global Address Space (PGAS) parallelization paradigm was introduced in an attempt to improve the clarity of the parallel implementation. We present our experiences of converting an unstructured high-resolution compressible Navier-Stokes CFD solver from MPI to PGAS Coarray Fortran. We present the challenges, methodology, and performance measurements of our approach using Coarray Fortran. With the Cray compiler, we observe Coarray Fortran as a viable alternative to MPI. We are hopeful that Intel and open-source implementations could be utilized in the future.

MPMD with Coarray Fortran (2008)

column Share on MPMD with Coarray Fortran (2008): an Example Program Author: Michael Siehl View Profile Authors Info & Claims ACM SIGPLAN Fortran ForumVolume 35Issue 2August 2016 pp 4–15https://doi.org/10.1145/2980025.2980026Published:29 July 2016Publication History 0citation98DownloadsMetricsTotal Citations0Total Downloads98Last 12 Months3Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

A general model checking framework for various memory consistency models

Relaxed memory consistency models are common and essential when multiple processes share a single global address space, such as when using multicore CPUs, distributed shared-memory programming languages, and so forth. Programming within these models is difficult and error prone, because of non-intuitive behaviors that could not occur in a strict consistency model. In addition, because the memory consistency models vary from language to language, and CPU to CPU, a program that may work correctly on one system may not work on another. To address the problem, this paper describes a model checking framework in which users are able to check their programs under various memory consistency models. More specifically, our framework provides a base model that exhibits very relaxed behaviors, and users are able to define various consistency models by adding constraints to the base model. This paper also describes McSPIN, a prototype implementation of a model checker based on the proposed framework. McSPIN can take a memory consistency model as an input, as well as a program and a property to be checked. We have specified the necessary constraints for three practical existing memory consistency models (Unified Parallel C, Coarray Fortran, and Itanium). McSPIN verified some example programs correctly, and confirmed the expected differences among the three models.

High-performance epistasis detection in quantitative trait GWAS

epiSNP is a program for identifying pairwise single nucleotide polymorphism (SNP) interactions (epistasis) in quantitative-trait genome-wide association studies (GWAS). A parallel MPI version (EPISNPmpi) was created in 2008 to address this computationally expensive analysis on large data sets with many quantitative traits and SNP markers. However, the falling cost of genotyping has led to an explosion of large-scale GWAS data sets that challenge EPISNPmpi’s ability to compute results in a reasonable amount of time. Therefore, we optimized epiSNP for modern multi-core and highly parallel many-core processors to efficiently handle these large data sets. This paper describes the serial optimizations, dynamic load balancing using MPI-3 RMA operations, and shared-memory parallelization with OpenMP to further enhance load balancing and allow execution on the Intel Xeon Phi coprocessor (MIC). For a large GWAS data set, our optimizations provided a 38.43× speedup over EPISNPmpi on 126 nodes using 2 MICs on TACC’s Stampede Supercomputer. We also describe a Coarray Fortran (CAF) version that demonstrates the suitability of PGAS languages for problems with this computational pattern. We show that the Coarray version performs competitively with the MPI version on the NERSC Edison Cray XC30 supercomputer. Finally, the performance benefits of hyper-threading for this application on Edison (average 1.35× speedup) are demonstrated.

A Monte Carlo-based Poisson's equation solver parallelized with Coarray Fortran

Poisson's equation is found in many scientific problems, such as heat transfer and electric field calculations. Although many different techniques are involved in solving Poisson's equation, we focused on the Monte Carlo method (MCM). We preferred the MCM not only because of its simple algorithm but also for its excellent parallel efficiency. Parallelization is one of the most effective techniques for reducing computation time. Among many parallelization paradigms, such as OpenMP (open multiprocessing), MPI (message passing interface), and PGAS (partitioned global address space), we adopted the PGAS-based Coarray Fortran (CAF). In this paper, we demonstrated that parallelization of Poisson's equation solver with CAF was quite painless. After parallelization, we solved Poisson's equation for a nonrectangular domain. We started with a workstation that consisted of 8 cores and we continued with a Cray supercomputer of up to 512 cores. The results of the parallel solvers were validated using exact solutions. We demonstrated that the error was less than 1.6%. Additionally, solution times and speedups of the CAF-based solver were compared with a solver that utilized MPI or OpenMP. OpenMP was not able to compete against CAF and MPI because of the "while" loop restriction. The CAF-based solver performed slightly better (7.5%) than the MPI provided that core numbers were between 2 and 32. However, CAF and MPI performed similarly for higher numbers of cores.

Leveraging OpenCoarrays to Support Coarray Fortran on IBM Power8E

We include a few comparisons between the performance a competent application developer would attain with two-sided MPI communication and the performance achieved with the newer MPI 3.0 one-sided communication that OpenCoarrays generates from CAF programs. We also demonstrate the OpenCoarrays support for the parallel collective subroutines that have been proposed for the upcoming Fortran 2015 standard.

Comparing Coarray Fortran (CAF) with MPI for several structured mesh PDE applications

Language-based approaches to parallelism have been incorporated into the Fortran standard. These Fortran extensions go under the name of Coarray Fortran (CAF) and full-featured compilers that support CAF have become available from Cray and Intel; the GNU implementation is expected in 2015. CAF combines elegance of expression with simplicity of implementation to yield an efficient parallel programming language. Elegance of expression results in very compact parallel code. The existence of a standard helps with portability and maintainability. CAF was designed to excel at one-sided communication and similar functions that support one-sided communication are also available in the recent MPI-3 standard. One-sided communication is expected to be very valuable for structured mesh applications involving partial differential equations, amongst other possible applications. This paper focuses on a comparison of CAF and MPI for a few very useful applications areas that are routinely used for solving partial differential equations on structured meshes. The three specific areas are Fast Fourier Techniques, Computational Fluid Dynamics, and Multigrid Methods.For each of those applications areas, we have developed optimized CAF code and optimized MPI code that is based on the one-sided messaging capabilities of MPI-3. Weak scalability studies that compare CAF and MPI-3 are presented on up to 65,536 processors. Both paradigms scale well, showing that they are well-suited for Petascale-class applications. Some of the applications shown (like Fast Fourier Techniques and Computational Fluid Dynamics) require large, coarse-grained messaging. Such applications emphasize high bandwidth. Our other application (Multigrid Methods) uses pointwise smoothers which require a large amount of fine-grained messaging. In such applications, a premium is placed on low latency. Our studies show that both CAF and MPI-3 offer the twin advantages of high bandwidth and low latency for messages of all sizes. Even for large numbers of processors, CAF either draws level with MPI-3 or shows a slight advantage over MPI-3. Both CAF and MPI-3 are shown to provide substantial advantages over MPI-2.In addition to the weak scalability studies, we also catalogue some of the best-usage strategies that we have found for our successful implementations of one-sided messaging in CAF and MPI-3. We show that CAF code is of course much easier to write and maintain, and the simpler syntax makes the parallelism easier to understand.

A Coarray Fortran implementation to support data-intensive application development

In this paper, we describe our experiences in implementing and applying Coarray Fortran (CAF) for the development of data-intensive applications in the domain of Oil and Gas exploration. The succes...

Portable, MPI-interoperable coarray fortran

The past decade has seen the advent of a number of parallel programming models such as Coarray Fortran (CAF), Unified Parallel C, X10, and Chapel. Despite the productivity gains promised by these models, most parallel scientific applications still rely on MPI as their data movement model. One reason for this trend is that it is hard for users to incrementally adopt these new programming models in existing MPI applications. Because each model use its own runtime system, they duplicate resources and are potentially error-prone. Such independent runtime systems were deemed necessary because MPI was considered insufficient in the past to play this role for these languages. The recently released MPI-3, however, adds several new capabilities that now provide all of the functionality needed to act as a runtime, including a much more comprehensive one-sided communication framework. In this paper, we investigate how MPI-3 can form a runtime system for one example programming model, CAF, with a broader goal of enabling a single application to use both MPI and CAF with the highest level of interoperability.

A Coarray Fortran implementation to support data-intensive application development

In this paper, we describe our experiences in implementing and applying Coarray Fortran (CAF) for the development of data-intensive applications in the domain of Oil and Gas exploration. The successful porting of reverse time migration (RTM), a data-intensive algorithm and one of the largest uses of computational resources in seismic exploration, is described, and results are presented demonstrating that the CAF implementation provides comparable performance to an equivalent MPI version. We also include a discussion on parallel I/O and how it may be incorporated into the CAF programming model.

A preliminary evaluation of the hardware acceleration of the Cray Gemini interconnect for PGAS languages and comparison with MPI

The Gemini interconnect on the Cray XE6 platform provides for lightweight remote direct memory access (RDMA) between nodes, which is useful for implementing partitioned global address space (PGAS) languages like UPC and Co-Array Fortran. In this paper, we perform a study of Gemini performance using a set of communication microbenchmarks and compare the performance of one-sided communication in PGAS languages with two-sided MPI. Our results demonstrate the performance benefits of the PGAS model on Gemini hardware, showing in what circumstances and by how much one-sided communication outperforms two-sided in terms of messaging rate, aggregate bandwidth, and computation and communication overlap capability. For example, for 8-byte and 2KB messages the one-sided messaging rate is 5 and 10 times greater respectively than the twosided one. The study also reveals important information about how to optimize one-sided Gemini communication.