One-sided Communication Research Articles

Implementing OpenSHMEM for the Adapteva Epiphany RISC Array Processor

The energy-efficient Adapteva Epiphany architecture exhibits massive many-core scalability in a physically compact 2D array of RISC cores with a fast network-on-chip (NoC). With fully divergent cores capable of MIMD execution, the physical topology and memory-mapped capabilities of the core and network translate well to partitioned global address space (PGAS) parallel programming models. Following an investigation into the use of two-sided communication using threaded MPI, one-sided communication using SHMEM is being explored. Here we present work in progress on the development of an OpenSHMEM 1.2 implementation for the Epiphany architecture.

HPCmatlab: A Framework for Fast Prototyping of Parallel Applications in Matlab

The HPCmatlab framework has been developed for Distributed Memory Programming in Matlab/Octave using the Message Passing Interface (MPI). The communication routines in the MPI library are implemented using MEX wrappers. Point-to-point, collective as well as one-sided communication is supported. Benchmarking results show better performance than the Mathworks Distributed Computing Server. HPCmatlab has been used to successfully parallelize and speed up Matlab applications developed for scientific computing. The application results show good scalability, while preserving the ease of programmability. HPCmatlab also enables shared memory programming using Pthreads and Parallel I/O using the ADIOS package.

Optimizing NEURON Simulation Environment Using Remote Memory Access with Recursive Doubling on Distributed Memory Systems.

Increase in complexity of neuronal network models escalated the efforts to make NEURON simulation environment efficient. The computational neuroscientists divided the equations into subnets amongst multiple processors for achieving better hardware performance. On parallel machines for neuronal networks, interprocessor spikes exchange consumes large section of overall simulation time. In NEURON for communication between processors Message Passing Interface (MPI) is used. MPI_Allgather collective is exercised for spikes exchange after each interval across distributed memory systems. The increase in number of processors though results in achieving concurrency and better performance but it inversely affects MPI_Allgather which increases communication time between processors. This necessitates improving communication methodology to decrease the spikes exchange time over distributed memory systems. This work has improved MPI_Allgather method using Remote Memory Access (RMA) by moving two-sided communication to one-sided communication, and use of recursive doubling mechanism facilitates achieving efficient communication between the processors in precise steps. This approach enhanced communication concurrency and has improved overall runtime making NEURON more efficient for simulation of large neuronal network models.

Leveraging OpenCoarrays to Support Coarray Fortran on IBM Power8E

We include a few comparisons between the performance a competent application developer would attain with two-sided MPI communication and the performance achieved with the newer MPI 3.0 one-sided communication that OpenCoarrays generates from CAF programs. We also demonstrate the OpenCoarrays support for the parallel collective subroutines that have been proposed for the upcoming Fortran 2015 standard.

Design of a Multithreaded Barnes-Hut Algorithm for Multicore Clusters

We describe in this paper an implementation of the Barnes-Hut algorithm on multicore clusters. Based on a partitioned global address space (PGAS) library, the design integrates intranode multithreading and internode one-sided communication, exemplifying a PGAS $+$ X programming style. Within a node, the computation is decomposed into tasks (subtasks) and multitasking is used to hide network latency. We study the tradeoffs between locality in private caches and locality in shared caches and bring the insights into the design. As a result, our implementation consumes less memory per core, invokes less internode communication, and enjoys better load-balancing strategies. The final code achieves up to 41 percent performance improvement over a non-multithreaded counterpart. Through detailed comparison, we also show its advantages over other well-known Barnes-Hut implementations, both in programming complexity and in performance.

Remote Memory Access Programming in MPI-3

The Message Passing Interface (MPI) 3.0 standard, introduced in September 2012, includes a significant update to the one-sided communication interface, also known as remote memory access (RMA). In particular, the interface has been extended to better support popular one-sided and global-address-space parallel programming models to provide better access to hardware performance features and enable new data-access modes. We present the new RMA interface and specify formal axiomatic models for data consistency and access semantics. Such models can help users reason about details of the semantics that are hard to extract from the English prose in the standard. It also fosters the development of tools and compilers, enabling them to automatically analyze, optimize, and debug RMA programs.

Automatic tuning of sparse matrix-vector multiplication on multicore clusters

To have good performance and scalability, parallel applications should be sophisticatedly optimized to exploit intra-node parallelism and reduce inter-node communication on multicore clusters. This paper investigates the automatic tuning of the sparse matrix-vector (SpMV) multiplication kernel implemented in a partitioned global address space language, which supports a hybrid thread- and process-based communication layer for multicore systems. One-sided communication is used for inter-node data exchange, while intra-node communication uses a mix of process shared memory and multithreading. We develop performance models to facilitate selecting the best configuration of threads and processes hybridization as well as the best communication pattern for SpMV. As a result, our tuned SpMV in the hybrid runtime environment consumes less memory and reduces inter-node communication volume, without damaging the data locality. Experiments are conducted on 12 real sparse matrices. On 16-node Xeon and 8-node Opteron clusters, our tuned SpMV kernel gets on average 1.4X and 1.5X improvement in performance over the well-optimized process-based message-passing implementation, respectively.

Comparing Coarray Fortran (CAF) with MPI for several structured mesh PDE applications

Language-based approaches to parallelism have been incorporated into the Fortran standard. These Fortran extensions go under the name of Coarray Fortran (CAF) and full-featured compilers that support CAF have become available from Cray and Intel; the GNU implementation is expected in 2015. CAF combines elegance of expression with simplicity of implementation to yield an efficient parallel programming language. Elegance of expression results in very compact parallel code. The existence of a standard helps with portability and maintainability. CAF was designed to excel at one-sided communication and similar functions that support one-sided communication are also available in the recent MPI-3 standard. One-sided communication is expected to be very valuable for structured mesh applications involving partial differential equations, amongst other possible applications. This paper focuses on a comparison of CAF and MPI for a few very useful applications areas that are routinely used for solving partial differential equations on structured meshes. The three specific areas are Fast Fourier Techniques, Computational Fluid Dynamics, and Multigrid Methods.For each of those applications areas, we have developed optimized CAF code and optimized MPI code that is based on the one-sided messaging capabilities of MPI-3. Weak scalability studies that compare CAF and MPI-3 are presented on up to 65,536 processors. Both paradigms scale well, showing that they are well-suited for Petascale-class applications. Some of the applications shown (like Fast Fourier Techniques and Computational Fluid Dynamics) require large, coarse-grained messaging. Such applications emphasize high bandwidth. Our other application (Multigrid Methods) uses pointwise smoothers which require a large amount of fine-grained messaging. In such applications, a premium is placed on low latency. Our studies show that both CAF and MPI-3 offer the twin advantages of high bandwidth and low latency for messages of all sizes. Even for large numbers of processors, CAF either draws level with MPI-3 or shows a slight advantage over MPI-3. Both CAF and MPI-3 are shown to provide substantial advantages over MPI-2.In addition to the weak scalability studies, we also catalogue some of the best-usage strategies that we have found for our successful implementations of one-sided messaging in CAF and MPI-3. We show that CAF code is of course much easier to write and maintain, and the simpler syntax makes the parallelism easier to understand.

Data decomposition in Monte Carlo neutron transport simulations using global view arrays

Accommodating large tally data can be a challenging problem for Monte Carlo neutron transport simulations. Current approaches include either simple data replication, or are based on application-controlled decomposition such as domain partitioning or client/server models, which are limited by either memory cost or performance loss. We propose and analyze an alternative solution based on global view arrays. By using global view arrays, tallies are naturally partitioned into small globally addressable blocks that fit in the limited on-node memory of compute nodes, achieving both highly scalable memory and performance efficiency. This approach also greatly simplifies the programmability compared with application-controlled approaches. Our implementation is based on integrating a global view library built on MPI one-sided communication, global view resilience (GVR), into the OpenMC Monte Carlo transport code. The remote memory access (RMA)-based global view array implementation is able to achieve 85% efficiency at 16,384 processes compared with 1,000 processes with 2.39 TB mesh tally across 1,366 nodes on a Cray XC30 supercomputer. Our results improve scalability significantly compared with the tally server approach and are better than any other published results, indicating that global view array is a promising alternative to enable full-core light water reactor analysis on current and future computer systems.

Asymmetric quantum network based on multipartite Einstein–Podolsky–Rosen steering

In this work, we propose an asymmetric multi-user access optical network with hierarchical structure in terms of steering ability. The N-partite system is generated by mixing two squeezed lights and N−2 vacuum modes at N−1 beam splitters. The asymmetry of this system can be constructed by choosing proper reflectivities of beam splitters and introducing asymmetric loss to channels of the network, such that the steering ability is distributed asymmetrically among users. By proper construction, one superior in the network possesses higher steering ability than any one of the remaining N−1 subordinates. We show how to accomplish the directional quantum steering between them, i.e., the superior can apparently steer any subordinate’s state, but it cannot always happen in the opposite direction. In addition, we quantify the thresholds of loss tolerance that will prevent the extraction of information of the superior by M(1≤M≤N−1) subordinates’ local measurements. An important feature of such a network is that the superior can send a secret message to certain subordinates without trustworthy assumptions about them and their apparatus. Our findings have applications in one-sided device-independent quantum communication protocol with multi-users.

Detection loophole attacks on semi-device-independent quantum and classical protocols

Semi-device-independent quantum protocols realize information tasks -- e.g. secure key distribution, random access coding, and randomness generation -- in a scenario where no assumption on the internal working of the devices used in the protocol is made, except their dimension. These protocols offer two main advantages: first, their implementation is often less demanding than fully-device-independent protocols. Second, they are more secure than their device-dependent counterparts. Their classical analogous is represented by random access codes, which provide a general framework for describing one-sided classical communication tasks. We discuss conditions under which detection inefficiencies can be exploited by a malicious provider to fake the performance of semi-device-independent quantum and classical protocols -- and how to prevent it.

경량 사물 인터넷 클라우드를 위한 DB 기반 가상 파일 시스템 구현

사물 인터넷은 개인, 산업, 공공 분야를 망라하여 각 분야의 말단 기기나 센서에 대한 직접 접근을 추구하는 상시 연결 인터넷 개념이다. 사물 인터넷에서는 실시간 데이터 접근이 가능하고, 디바이스의 데이터 포맷과 구성 형태가 다양하다. 또한, 사물 인터넷 환경에서는 기기 제어를 위한 양방향 통신이 이루어지기도 한다. 이런 관점에서, 사물 인터넷은 주로 인간이 사용하는 데스크톱에서 생성된 데이터가 간단하고 일방적인 인터넷 통신을 통해 서버에 집중되는 기존 인터넷과 대비된다. 이러한 사물 인터넷의 클라우드 혹은 포털 서비스를 위해서는 분산된 디바이스들에 대한 체계적인 명명 서비스와 이들의 다양성을 포용할 수 있는 일관된 접근 인터페이스를 제공하는 파일 관리 프레임이 요구된다. 이 연구에서는 사물 인터넷 구성요소의 속성 정보를 유닉스 스타일의 파일 시스템 뷰로 관리하는 DB 기반 가상 파일 시스템을 구현한다. 구현된 가상 쉘에 로그인한 사용자들은 가상 파일 시스템을 항해함으로써 사물 인터넷 구성요소를 탐색할 수 있고, 유닉스 스타일의 파일 입 출력 API를 이용하여 디바이스에 직접 접근할 수 있다. 구현된 DB 기반 가상 파일 시스템은 디렉터리 체계와 분산된 데이터 및 디바이스에 대한 지시자만을 관리하므로 가볍고 유연하다. Mkdir(), chdir() 등 구현된 주요 가상 쉘 프리미티브에 대한 시험 결과, 구현된 가상 파일 시스템의 원활한 기능이 검증되었다. 그리고 구현 파일 시스템에 간단한 디렉터리 캐시 기법을 적용한 경우 윈도우 운영체제 파일 시스템보다 더 우수한 탐색 성능을 보였다. IoT(Internet of Things) is a concept of connected internet pursuing direct access to devices or sensors in fused environment of personal, industrial and public area. In IoT environment, it is possible to access realtime data, and the data format and topology of devices are diverse. Also, there are bidirectional communications between users and devices to control actuators in IoT. In this point, IoT is different from the conventional internet in which data are produced by human desktops and gathered in server systems by way of one-sided simple internet communications. For the cloud or portal service of IoT, there should be a file management framework supporting systematic naming service and unified data access interface encompassing the variety of IoT things. This paper implements a DB-based virtual file system maintaining attributes of IoT things in a UNIX-styled file system view. Users who logged in the virtual shell are able to explore IoT things by navigating the virtual file system, and able to access IoT things directly via UNIX-styled file I O APIs. The implemented virtual file system is lightweight and flexible because it maintains only directory structure and descriptors for the distributed IoT things. The result of a test for the virtual shell primitives such as mkdir() or chdir() shows the smooth functionality of the virtual file system, Also, the exploring performance of the file system is better than that of Window file system in case of adopting a simple directory cache mechanism.

Scalable PGAS collective operations in NUMA clusters

The increasing number of cores per processor is turning manycore-based systems in pervasive. This involves dealing with multiple levels of memory in non uniform memory access (NUMA) systems and processor cores hierarchies, accessible via complex interconnects in order to dispatch the increasing amount of data required by the processing elements. The key for efficient and scalable provision of data is the use of collective communication operations that minimize the impact of bottlenecks. Leveraging one sided communications becomes more important in these systems, to avoid unnecessary synchronization between pairs of processes in collective operations implemented in terms of two sided point to point functions. This work proposes a series of algorithms that provide a good performance and scalability in collective operations, based on the use of hierarchical trees, overlapping one-sided communications, message pipelining and the available NUMA binding features. An implementation has been developed for Unified Parallel C, a Partitioned Global Address Space language, which presents a shared memory view across the nodes for programmability, while keeping private memory regions for performance. The performance evaluation of the proposed implementation, conducted on five representative systems (JuRoPA, JUDGE, Finis Terrae, SVG and Superdome), has shown generally good performance and scalability, even outperforming MPI in some cases, which confirms the suitability of the developed algorithms for manycore architectures.

Discrete Event Execution with One-Sided and Two-Sided GVT Algorithms on 216,000 Processor Cores

Global Virtual Time (GVT) computation is a key determinant of the efficiency and runtime dynamics of Parallel Discrete Event Simulations (PDES), especially on large-scale parallel platforms. Here, three execution modes of a generalized GVT computation algorithm are studied on high-performance parallel computing systems: (1) a synchronous GVT algorithm that affords ease of implementation, (2) an asynchronous GVT algorithm that is more complex to implement but can relieve blocking latencies, and (3) a variant of the asynchronous GVT algorithm to exploit one-sided communication in extant supercomputing platforms. Performance results are presented of implementations of these algorithms on up to 216,000 cores of a Cray XT5 system, exercised on a range of parameters: optimistic and conservative synchronization, fine- to medium-grained event computation, synthetic and nonsynthetic applications, and different lookahead values. Detailed PDES-specific runtime metrics are presented to further the understanding of tightly coupled discrete event dynamics on massively parallel platforms.

Portable, MPI-interoperable coarray fortran

The past decade has seen the advent of a number of parallel programming models such as Coarray Fortran (CAF), Unified Parallel C, X10, and Chapel. Despite the productivity gains promised by these models, most parallel scientific applications still rely on MPI as their data movement model. One reason for this trend is that it is hard for users to incrementally adopt these new programming models in existing MPI applications. Because each model use its own runtime system, they duplicate resources and are potentially error-prone. Such independent runtime systems were deemed necessary because MPI was considered insufficient in the past to play this role for these languages. The recently released MPI-3, however, adds several new capabilities that now provide all of the functionality needed to act as a runtime, including a much more comprehensive one-sided communication framework. In this paper, we investigate how MPI-3 can form a runtime system for one example programming model, CAF, with a broader goal of enabling a single application to use both MPI and CAF with the highest level of interoperability.

POSH: Paris OpenSHMEM A High-performance OpenSHMEM Implementation for Shared Memory Systems

In this paper we present the design and implementation of POSH, an Open-Source implemen- tation of the OpenSHMEM standard. We present a model for its communications, and prove some properties on the memory model defined in the OpenSHMEM specification. We present some performance measurements of the communication library featured by POSH and com- pare them with an existing one-sided communication library. POSH can be downloaded from http://www.lipn.fr/∼coti/POSH.

PARALLEL SIMULATION OF LARGE-SCALE ARTIFICIAL SOCIETY WITH GPU AS COPROCESSOR

Large-scale artificial societies with millions or billions of agents call for high-performance parallel simulation. Prevailing supercomputers with thousands of CPUs and GPUs make it possible to carry out such simulation. The key is to distribute large-scale agents to massive cores of CPUs and GPUs properly for parallel computing with efficient communication and synchronization. For simplicity and efficiency, a modified discrete event system specification (DEVS) is proposed for large-scale artificial society modeling and parallelism is exploited in agent models because similar agents usually share similar behaviors. Through phased synchronization, a two-tier parallel simulation engine is designed with support of MPI and OpenCL where GPU is used as coprocessor. One-sided communication is used for reflection of remote simulation objects and message passing between processes. A general kernel function prototype is elaborately designed and conditionally compiled for execution on both CPU and GPU. An artificial society for epidemic study is used to test the performance on a supercomputer with 1024 CPU cores and 1792 GPU cores. The speedup reaches 3512 for even 2 billion agents with GPU acceleration which is far over 701 when only CPUs are used. It turns out feasible for parallel simulation of large-scale artificial society with GPU as coprocessor.

MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared memory

Hybrid parallel programming with the message passing interface (MPI) for internode communication in conjunction with a shared-memory programming model to manage intranode parallelism has become a dominant approach to scalable parallel programming. While this model provides a great deal of flexibility and performance potential, it saddles programmers with the complexity of utilizing two parallel programming systems in the same application. We introduce an MPI-integrated shared-memory programming model that is incorporated into MPI through a small extension to the one-sided communication interface. We discuss the integration of this interface with the MPI 3.0 one-sided semantics and describe solutions for providing portable and efficient data sharing, atomic operations, and memory consistency. We describe an implementation of the new interface in the MPICH2 and Open MPI implementations and demonstrate an average performance improvement of 40 % to the communication component of a five-point stencil solver.

On eliminating synchronous communication in molecular simulations to improve scalability

Molecular dynamics simulation, as a complementary tool to experimentation, has become an important methodology for the understanding and design of molecular systems as it provides access to properties that are difficult, impossible or prohibitively expensive to obtain experimentally. Many of the available software packages have been parallelized to take advantage of modern massively concurrent processing resources. The challenge in achieving parallel efficiency is commonly attributed to the fact that molecular dynamics algorithms are communication intensive.This paper illustrates how an appropriately chosen data distribution and asynchronous one-sided communication approach can be used to effectively deal with the data movement within the Global Arrays/ARMCI programming model framework. A new put_notify capability is presented here, allowing the implementation of the molecular dynamics algorithm without any explicit global or local synchronization or global data reduction operations. In addition, this push-data model is shown to very effectively allow hiding data communication behind computation. Rather than data movement or explicit global reductions, the implicit synchronization of the algorithm becomes the primary challenge for scalability. Without any explicit synchronous operations, the scalability of molecular simulations is shown to depend only on the ability to evenly balance computational load.

Two-sided communication in competing mechanism games

We consider multiple-principal multiple-agent games of incomplete information. We identify a class of two-way communication mechanisms which mirror those considered in the single principal analysis of Myerson (1982). In such mechanisms, every agent truthfully reveals her type to all principals, and obeys the private recommendations she receives from each of them. We show that there is a rationale in restricting attention to this class of mechanisms: if principals use these mechanisms, there is no unilateral incentive to deviate towards more sophisticated ones. We develop three examples to analyze possible extensions and limits of our approach. The first two examples show that the restriction to direct and incentive compatible mechanisms is not sufficient to provide a complete characterization of all pure strategy equilibria. The third one shows that private recommendations play a fundamental role in competing mechanism games, suggesting that one cannot safely restrict to one-sided communication mechanisms.