Distributed Shared Memory Research Articles

Parallel Processing Distributed-Memory Approach Influences on Performance of Multicomputer-Multicore Systems Using Single-Process Single-Thread

Based on client/server architecture concepts, this research suggests a method for creating a multicomputer-multicore distributed memory system that can be implemented on distributed-shared memory systems. Both of number of the participated computers and number of existed processors for each of these computers, this research was depended with the specific design and its implementation. The suggested system has two primary phases: monitoring and managing the programmes that may be executed on multiple distributed-multi-core architectures with (2, 4, and 8) CPUs to perform a certain job. There might be a single client and unlimited servers in the network. The implementation phase relies on three separate scenarios covering most of the design space. The suggested system can determine the start time, duration, CPU use, kernel time, user time, waiting time, and end time for each server in the system. Single-Process Single-Thread (SPST) is considered a possible situation while developing User Programmes (UPs). The findings confirmed that more processing power (more servers and more processors on each server) increases the speed at which tasks can be solved. There was a 2.877-fold gain in task processing speed after considering three different possible SPST UPs situations. The C# programming language is used to create this system.

Distributed Shared Memory : A Page-Based Approach

In 1986, Kai Li's PhD thesis, "Shared Virtual Memory on Loosely Coupled Microprocessors," introduced the concept of Distributed Shared Memory (DSM) systems, which has since facilitated further research in this field. One approach to implementing DSM in distributed systems is the page-based method, which utilizes virtual memory techniques to map pages of a process' address space onto the physical memory of multiple nodes. The system automatically moves pages between nodes as needed, ensuring coherence and consistency for shared page updates. Page-based DSM offers several advantages, including improved performance due to reduced communication overhead and better data locality, simplified programming models that resemble centralized systems, and improved scalability by allowing additional nodes to join the system and expand memory capacity. However, its suitability for certain distributed systems depends on their specific requirements and characteristics. Page-based DSM has been applied in various distributed systems, including high- performance computing, cloud computing, and distributed databases. Nonetheless, remote memory access overheads, cache coherence issues, and the necessity of a well-connected network remain potential limitations of this approach. Key words: Distributed memory, Shared memory, Paging, Virtual memory, memory coherence, memory consistency

GiantVM: A Novel Distributed Hypervisor for Resource Aggregation with DSM-aware Optimizations

We present GiantVM, 1 an open-source distributed hypervisor that provides the many-to-one virtualization to aggregate resources from multiple physical machines. We propose techniques to enable distributed CPU and I/O virtualization and distributed shared memory (DSM) to achieve memory aggregation. GiantVM is implemented based on the state-of-the-art type-II hypervisor QEMU-KVM, and it can currently host conventional OSes such as Linux. (1) We identify the performance bottleneck of GiantVM to be DSM, through a top-down performance analysis. Although GiantVM offers great opportunities for CPU-intensive applications to enjoy the aggregated CPU resources, memory-intensive applications could suffer from cross-node page sharing, which requires frequent DSM involvement and leads to performance collapse. We design the guest-level thread scheduler, DaS (DSM-aware Scheduler), to overcome the bottleneck. When benchmarking with NAS Parallel Benchmarks, the DaS could achieve a performance boost of up to 3.5×, compared to the default Linux kernel scheduler. (2) While evaluating DaS, we observe the advantage of GiantVM as a resource reallocation facility. Thanks to the SSI abstraction of GiantVM, migration could be done by guest-level scheduling. DSM allows standby pages in the migration destination, which need not be transferred through the network. The saved network bandwidth is 68% on average, compared to VM live migration. Resource reallocation with GiantVM increases the overall CPU utilization by 14.3% in a co-location experiment.

Exploring a Hybrid Voting-based Eviction Policy for Caches and Sparse Directories on Manycore Architectures

In manycore systems, eviction decisions related to caches and memory coherence greatly impact system performance, thereby emphasizing their importance. Extensive research has produced numerous standalone eviction policies such as LRU, LFU, FIFO, etc. all aiming to attain the Bélády optimum solution. Standalone eviction policies optimize for a single attribute (recency, frequency, etc.), limiting their impact on applications exhibiting non-uniform memory access patterns. The Hybrid Voting-based Eviction Policy (HyVE) extends multiple standalone eviction policies with a ranking system and evaluates them using concepts from the voting theory domain. The goal of HyVE is to make better replacement decisions by creating a consensus among its constituent eviction policies. With its inherent voting properties, HyVE takes different replacement decisions compared to its standalone counterparts, making it a unique and new eviction policy. We deploy and evaluate HyVE as part of two case-studies: last-level cache replacement decisions in a generic manycore environment, and sparse directory eviction decisions on a tile-based distributed shared memory (DSM) architecture. We explore different variants of HyVE, and evaluate them using workloads from the PARSEC and SPLASH-2 benchmark suites in a simulation environment. We also compare HyVE to state-of-the-art set-dueling and learning-based eviction policies. For last-level cache replacement decisions, HyVE reduces cache misses by 7.4% compared to the LRU policy, whereas DRRIP and Hawkeye reduce cache misses by 5.5% and 9.2% respectively compared to the LRU policy. Though Hawkeye exhibits better performance on average, HyVE offers a unique advantage for certain workloads by using a voting-based approach to solve the replacement problem. For sparse directory eviction decisions, results show that HyVE reduces coherence traffic and execution time by up to 11% compared to the LRU policy. We have synthesized HyVE on an FPGA prototype. Hardware analysis results show that HyVE’s constituent policies contribute the most to its overheads, while HyVE’s ranking and voting extensions do not add significant overheads. Timing analysis results show that HyVE’s logic delay is comparable to that of standalone eviction policies. Lastly, we evaluate HyVE on the FPGA prototype using characteristic micro-benchmarks that further emphasize HyVE’s ability to remain agnostic to varying data access patterns.

PsmArena: Partitioned shared memory for NUMA-awareness in multithreaded scientific applications

The Distributed Shared Memory (DSM) architecture is widely used in today's computer design to mitigate the ever-widening processing-memory gap, and it inevitably exhibits Non-Uniform Memory Access (NUMA) to shared-memory parallel applications. Failure to adapt to the NUMA effect can significantly downgrade application performance, especially on today's manycore platforms with tens to hundreds of cores. However, traditional approaches such as first-touch and memory policy fall short in false page-sharing, fragmentation, or ease of use. In this paper, we propose a partitioned shared-memory approach that allows multithreaded applications to achieve full NUMA-awareness with only minor code changes and develop an accompanying NUMA-aware heap manager which eliminates false page-sharing and minimizes fragmentation. Experiments on a 256-core cc-NUMA computing node show that the proposed approach helps applications to adapt to NUMA with only minor code changes and improves the performance of typical multithreaded scientific applications by up to 4.3 folds with the increased use of cores.

Load balanced distributed directories

We present LB-Spiral, a novel distributed directory protocol for shared objects, suitable for large-scale distributed shared memory systems. Each shared object has an owner node that can modify its value. The ownership may change by moving the object from one node to another in response to move requests. The value of an object can be read by other nodes with lookup requests. The distinctive feature of LB-Spiral is that it balances the processing load on nodes in addition to minimizing the communication cost in general network topologies. In contrast, the existing distributed directory protocols for general network topologies only minimize the communication cost. In particular, LB-Spiral achieves poly-log approximation for both load and communication cost in general networks with respect to the problem parameters. Simulation results show that the established theoretical results translate well in practice.

Random Priority-Based Thrashing Control for Distributed Shared Memory

Shared memory is widely used for inter-process communication. The shared memory abstraction allows computation to be decoupled from communication, which offers benefits, including portability and ease of programming. To enable shared memory access by processes that are on different machines, distributed shared memory (DSM) can be employed. However, DSM systems can suffer from thrashing: while different processes update certain hot data items, the largest amount of effort is spent on data synchronization, and little progress is made by each process. To avoid interference between processes during data updating while providing shared memory at page granularity, more time is reserved for a writer to hold a page in a traditional manner. In this paper, we report on complex thrashing, which can explain why extending the time of holding a page might not be sufficient to control thrashing. To increase the throughput, we propose a thrashing control mechanism that allows each process to update a set of pages during a period of time, where the pages compose a logical area. Because of the isolation of areas, updates on different areas can be performed concurrently. To allow the areas to be fairly well used, each process is assigned with a random priority for thrashing control. The thrashing control mechanism is implemented on a Linux-based DSM system. Performance results show that the execution time of the applications that are apt to cause system thrashing can be significantly reduced by our approach.

Automation in Distributed Shared Memory Testing for Multi-Processor Systems

This research paper explores the critical domain of automated testing for Distributed Shared Memory (DSM) systems in multi-processor environments. As the complexity of multi-core and distributed computing systems continues to grow, ensuring the reliability and performance of DSM implementations becomes increasingly challenging. This study investigates various automated testing strategies, including test generation techniques, fault injection mechanisms, and concurrency detection methods. It also examines automated test execution frameworks, real-time monitoring solutions, and advanced verification and validation techniques. The research highlights the challenges faced in DSM testing, such as scalability issues and non-determinism, and proposes future directions for research, including the integration of artificial intelligence and cloud-based testing platforms. The findings of this study contribute to the advancement of DSM testing methodologies and provide valuable insights for both researchers and practitioners in the field of distributed systems and parallel computing.

Scaling out NUMA-Aware Applications with RDMA-Based Distributed Shared Memory

The multicore evolution has stimulated renewed interests in scaling up applications on shared-memory multiprocessors, significantly improving the scalability of many applications. But the scalability is limited within a single node; therefore programmers still have to redesign applications to scale out over multiple nodes. This paper revisits the design and implementation of distributed shared memory (DSM) as a way to scale out applications optimized for non-uniform memory access (NUMA) architecture over a well-connected cluster. This paper presents MAGI, an efficient DSM system that provides a transparent shared address space with scalable performance on a cluster with fast network interfaces. MAGI is unique in that it presents a NUMA abstraction to fully harness the multicore resources in each node through hierarchical synchronization and memory management. MAGI also exploits the memory access patterns of big-data applications and leverages a set of optimizations for remote direct memory access (RDMA) to reduce the number of page faults and the cost of the coherence protocol. MAGI has been implemented as a user-space library with pthread-compatible interfaces and can run existing multithreaded applications with minimized modifications. We deployed MAGI over an 8-node RDMAenabled cluster. Experimental evaluation shows that MAGI achieves up to 9.25x speedup compared with an unoptimized implementation, leading to a scalable performance for large-scale data-intensive applications.

A new multi-resolution parallel framework for SPH

In this paper we present a new multi-resolution parallel framework, which is designed for large-scale SPH simulations of fluid dynamics. An adaptive rebalancing criterion and monitoring system is developed to integrate the CVP partitioning method as rebalancer to achieve dynamic load balancing of the system. A localized nested hierarchical data structure is developed in cooperation with a tailored parallel fast-neighbor-search algorithm to handle problems with arbitrarily adaptive smoothing-length and to construct ghost buffer particles in remote processors. The concept of “diffused graph” is proposed in this paper to improve the performance of the graph-based communication strategy. By utilizing the hybrid parallel model, the framework is able to exploit the full parallel potential of current state-of-the-art clusters based on Distributed Shared Memory (DSM) architectures. A range of gas dynamics benchmarks are investigated to demonstrate the capability of the framework and its unique characteristics. The performance is assessed in detail through intensive numerical experiments at various scales.

A mathematical model to calculate real cost/performance in software distributed shared memory on computing environments

One of the important factors in high-performance computing (HPC) is the cost/performance ratio. Calculation of cost/performance ratio is the main criterion for the separation of hardware computing systems (supercomputers) from software computing systems (Cluster, Grid, Peer-to-Peer). There are various economic methods to calculate hardware cost. In addition, there are numerous methods in software engineering to calculate the cost of developing and programming the scientific and engineering software. The computing power in the aforementioned systems is basically calculated with programs like LINPACK and HPCL. The inter-process communication is considered as a variable in calculating the cost of executing the scientific programs, whose nature and amount depends on the program execution itself. As there is a high dependency of effective variables in cost calculation of inter-process communications during the program execution, it should be used for calculating the cost of any application. This paper complements the existing methods by presenting a more comprehensive and accurate method to calculate the real cost of distributed shared memory (DSM) mechanisms used by HPC Systems. Therefore, a systematic method has been used to achieve a whole equation for DSM costing, determine the effective factors of the cost, and propose a method based on costing economic methods. Effective parameters are classified into two groups, namely DSM-inhere dependent and application-specific dependent parameters. Each parameter is presented and discussed, and the correlation between them specifies the system's weight on DSM real cost according to which the cost is modeled and validated analytically.

A real-time operating system supporting distributed shared memory for embedded control systems

The paper presents a real-time operating system (RTOS) that provides a distributed shared memory (DSM) service for distributed embedded control systems. Model-based design is widely adopted in embedded control software design and the source code of software modules can be generated from a controller model. The generated software modules exchange their input and output values through shared variables. We develop a RTOS with a DSM service to provide a location-transparent environment, in which distributed software modules can exchange input and output values through the DSM. The RTOS is an extension to OSEK OS. We use a real-time network called FlexRay, which is based on a time division multiple access (TDMA) protocol. The consistency of the DSM is maintained according to the order of data transfer through FlexRay, not using inter-node synchronization. The worst case response time of the DSM is predictable if the FlexRay communication is well configured.

Improving Performance of Distributed Shared Memory (DSM) on Multiprocessor Framework with Software Approach

Objectives: To design Distributed Shared Memory (DSM) for the multiprocessor distributed framework using a different software parametric approach that provides significant performance improvement against convention software based architectures. Methods/Statistical Analysis: Software distributed shared memory can be architected by using a different concept of an operating system, by utilizing a programming library and by extending underlying virtual address space architecture. It incorporates various design options like granularity, consistency model, implementation level, data organization, algorithms, protocols, etc. We have proposed few software parameter choices and impact which gives significant performance improvement compared to past designs to manage software distributed shared memory. This paper also discusses various issues that exist while moving toward software distributed shared memory implementation. Findings: There are two methodologies by which it is possible to achieve distributed shared memory design are first in hardware like cache coherence circuits and network interfaces and the second is software. Here the proposed system architecture makes major impact on programming, performance, design and cost. An algorithm is designed such a unique manner which resides in memory controller and make efficient global virtual memories. It is using variable as granularity which are shared that is more flexible for complex data structure and large database. It is defined using unique identifier which makes its mapping and retrieval more manageable using proposed consistency mechanism. Application/Improvements: Distributed shared memory optimization is a most important area of improving distributed system performance. By taking care of good choice on underlying issues and according to system’s design requirement, it possible to gain advantages of improved architecture which can be more used for various distributed applications where shared data plays a major role.

Improving Performance of Distributed Shared Memory (DSM) on Multiprocessor Framework with Software Approach

Improving Performance of Distributed Shared Memory (DSM) on Multiprocessor Framework with Software Approach

The Unicorn Runtime: Efficient Distributed Shared Memory Programming for Hybrid CPU-GPU Clusters

Programming hybrid CPU-GPU clusters is hard. This paper addresses this difficulty and presents the design and runtime implementation of Unicorn —a parallel programming model for hybrid CPU-GPU clusters. In particular, this paper proves that efficient distributed shared memory style programing is possible and its simplicity can be retained across CPUs and GPUs in a cluster, minus the frustration of dealing with race conditions. Further, this can be done with a unified abstraction, avoiding much of the complication of dealing with hybrid architectures. This is achieved with the help of transactional semantics (on shared global address spaces), deferred bulk data synchronization, workload pipelining and various communication and computation scheduling optimizations. We describe the said abstraction, our computation and communication scheduling system and report its performance on a few benchmarks like Matrix Multiplication , LU Decomposition and 2D FFT . We find that parallelization of coarse-grained applications like matrix multiplication or 2D FFT using our system requires only about 30 lines of C code to set up the runtime. The rest of the application code is regular single CPU/GPU implementation. This indicates the ease of extending parallel code to a distributed environment. The execution is efficient as well. When multiplying two square matrices of size $65,536\times 65,536$ , Unicorn achieves a peak performance of 7.88 TFlop/s when run over a cluster of 14 nodes with each node equipped with two Tesla M2070 GPUs and two 6-core Intel Xeon 2.67 GHz CPUs, connected over a 32 Gbps Infiniband network. In this paper, we also demonstrate that the Unicorn programming model can be efficiently used to implement high level abstractions like MapReduce. We use such an extension to implement PageRank and report its performance. For a sample web of 500 million web pages, our implementation completes a page rank iteration in about 18 seconds (on average) on a 14-node cluster.

A general and efficient divide-and-conquer algorithm framework for multi-core clusters

Divide-and-conquer is one of the most important patterns of parallelism, being applicable to a large variety of problems. In addition, the most powerful parallel systems available nowadays are computer clusters composed of distributed-memory nodes that contain an increasing number of cores that share a common memory. The optimal exploitation of these systems often requires resorting to a hybrid model that mimics the underlying hardware by combining a distributed and a shared memory parallel programming model. This results in longer development times and increased maintenance costs. In this paper we present a very general skeleton library that allows to parallelize any divide-and-conquer problem in hybrid distributed-shared memory systems with little effort while providing much flexibility and good performance. Our proposal combines a message-passing paradigm at the process level and a threaded model inside each process, hiding the related complexity from the user. The evaluation shows that this skeleton provides performance comparable, and often better than that of manually optimized codes while requiring considerably less effort when parallelizing applications on multi-core clusters.

A Distributed Shared Memory Model and C++ Templated Meta-Programming Interface for the Epiphany RISC Array Processor

The Adapteva Epiphany many-core architecture comprises a scalable 2D mesh Network-on-Chip (NoC) of low-power RíSC cores with minimal uncore functionality. Whereas such a processor offers high computational energy efficiency and parallel scalability, developing effective programming models that address the unique architecture features has presented many challenges. We present here a distributed shared memory (DSM) model supported in software transparently using C++ templated meta-programming techniques. The approach offers an extremely simple parallel programming model well suited for the architecture. Initial results are presented that demonstrate the approach and provide insight into the efficiency of the programming model and also the ability of the NoC to support a DSM without explicit control over data movement and localization.

Architecture, Languages, Compilation and Hardware support for Emerging ManYcore systems (ALCHEMY): Preface

Manycore systems are one of the key enabler technologies for most of current computational paradigms, including Internet of things, data-centers on chip and big data processing. These paradigms are characterized by tight and demanding requirements such as code portability, dynamicity, high performance, usability, predictability, reliability, low power and security.This combination of requirements has led to heterogeneous manycore systems which are extremely challenging to design and to program. As a result, a large body of research has focused on development of languages, simulation environments and analysis tools that allow to model and predict the behavior of this type of systems from early design stages.ALCHEMY 2017 presents five research works addressing the challenges of code portability, high performance, usability, security and reliability in manycore systems. These are namely:1. ”An OpenMP backend for the Sigma-C streaming language, addresses the software portability challenge of manycore architectures. It proposes an implementation of an OpenMP backend for the SigmaC language, a cycle-static data flow abstraction to program many-core embedded platforms. Its compilation scheme allows for utilization of future manycore embedded systems such as Kalray’s MPPA.2. A multi-level optimization strategy to improve the performance of the stencil computation combines manual vectorization, space tiling and stencil composition for achieving high performance of stencil kernels on manycore systems. The evaluation with three compilers (Intel, Clang and GCC) and two target multi-core platforms (Intel Broadwell and Ivybridge) reports better results compared to the state of the art.3. A Distributed Shared Memory Model and C++ Templated Meta-Programming Interface for the Epiphany RISC Array Processor addresses the usability challenge. It proposes techniques for data layout and parallel loop order abstraction as a parallel programming API targeting the Epiphany architecture. This results into a transparent distributed shared memory (DSM) model for Epiphany that eliminates the need to manage local data movement between cores.4. Towards Protected MPSoC Communication for Information Protection against a Malicious NoC deals with vulnerabilities on Network-on-Chip (NoC). The authors propose a security protocol which allows the secure communication among the cores of the system, even in the presence of Trojan insertions at the NoC whose aim is to modify and steal data.5. GPU-Accelerated Real-Time Path Planning and the Predictable Execution Model addresses the reliability challenge and tackles the important problem of ensuring reliable Worst Case Execution Time for Real-Time and Cyber Physical Systems. While considering heterogeneous (CPU/GPU), the idea is to separate memory and processor operations through Time-Division Multiplexing (TDM).

On the improvement of a scalable sparse direct solver for unsymmetrical linear equations

This paper focuses on the application level improvements in a sparse direct solver specifically used for large-scale unsymmetrical linear equations resulting from unstructured mesh discretization of coupled elliptic/hyperbolic PDEs. Existing sparse direct solvers are designed for distributed server systems taking advantage of both distributed memory and processing units. We conducted extensive numerical experiments with three state-of-the-art direct linear solvers that can work on distributed-memory parallel architectures; namely, MUMPS (MUMPS solver website, http://graal.ens-lyon.fr/MUMPS), WSMP (Technical Report TR RC-21886, IBM, Watson Research Center, Yorktown Heights, 2000), and SUPERLU_DIST (ACM Trans Math Softw 29(2):110---140, 2003). The performance of these solvers was analyzed in detail, using advanced analysis tools such as Tuning and Analysis Utilities (TAU) and Performance Application Programming Interface (PAPI). The performance is evaluated with respect to robustness, speed, scalability, and efficiency in CPU and memory usage. We have determined application level issues that we believe they can improve the performance of a distributed-shared memory hybrid variant of this solver, which is proposed as an alternative solver [SuperLU_MCDT (Many-Core Distributed)] in this paper. The new solver utilizing the MPI/OpenMP hybrid programming is specifically tuned to handle large unsymmetrical systems arising in reservoir simulations so that higher performance and better scalability can be achieved for a large distributed computing system with many nodes of multicore processors. Two main tasks are accomplished during this study: (i) comparisons of public domain solver algorithms; existing state-of-the-art direct sparse linear system solvers are investigated and their performance and weaknesses based on test cases are analyzed, (ii) improvement of direct sparse solver algorithm (SuperLU_MCDT) for many-core distributed systems is achieved. We provided results of numerical tests that were run on up to 16,384 cores, and used many sets of test matrices for reservoir simulations with unstructured meshes. The numerical results showed that SuperLU_MCDT can outperform SuperLU_DIST 3.3 in terms of both speed and robustness.

Design and Cost Analysis for a Fault-Tolerant Distributed Shared Memory System

Algorithms implementing distributed shared memory (DSM) were developed for ensuring consistency. The performance of DSM algorithms is dependent on system and usage parameters. However, ensuring these algorithms to tolerate faults is a problem that needs to be researched. In this study, we proposed fault-tolerant scheme for DSM system and analyzed reliability and fault-tolerant overhead. Using our analysis, we can choose a proper algorithm for DSM on error prone environment.