Distributed Shared Memory Systems Research Articles

Parallel Processing Distributed-Memory Approach Influences on Performance of Multicomputer-Multicore Systems Using Single-Process Single-Thread

Based on client/server architecture concepts, this research suggests a method for creating a multicomputer-multicore distributed memory system that can be implemented on distributed-shared memory systems. Both of number of the participated computers and number of existed processors for each of these computers, this research was depended with the specific design and its implementation. The suggested system has two primary phases: monitoring and managing the programmes that may be executed on multiple distributed-multi-core architectures with (2, 4, and 8) CPUs to perform a certain job. There might be a single client and unlimited servers in the network. The implementation phase relies on three separate scenarios covering most of the design space. The suggested system can determine the start time, duration, CPU use, kernel time, user time, waiting time, and end time for each server in the system. Single-Process Single-Thread (SPST) is considered a possible situation while developing User Programmes (UPs). The findings confirmed that more processing power (more servers and more processors on each server) increases the speed at which tasks can be solved. There was a 2.877-fold gain in task processing speed after considering three different possible SPST UPs situations. The C# programming language is used to create this system.

Load balanced distributed directories

We present LB-Spiral, a novel distributed directory protocol for shared objects, suitable for large-scale distributed shared memory systems. Each shared object has an owner node that can modify its value. The ownership may change by moving the object from one node to another in response to move requests. The value of an object can be read by other nodes with lookup requests. The distinctive feature of LB-Spiral is that it balances the processing load on nodes in addition to minimizing the communication cost in general network topologies. In contrast, the existing distributed directory protocols for general network topologies only minimize the communication cost. In particular, LB-Spiral achieves poly-log approximation for both load and communication cost in general networks with respect to the problem parameters. Simulation results show that the established theoretical results translate well in practice.

Random Priority-Based Thrashing Control for Distributed Shared Memory

Shared memory is widely used for inter-process communication. The shared memory abstraction allows computation to be decoupled from communication, which offers benefits, including portability and ease of programming. To enable shared memory access by processes that are on different machines, distributed shared memory (DSM) can be employed. However, DSM systems can suffer from thrashing: while different processes update certain hot data items, the largest amount of effort is spent on data synchronization, and little progress is made by each process. To avoid interference between processes during data updating while providing shared memory at page granularity, more time is reserved for a writer to hold a page in a traditional manner. In this paper, we report on complex thrashing, which can explain why extending the time of holding a page might not be sufficient to control thrashing. To increase the throughput, we propose a thrashing control mechanism that allows each process to update a set of pages during a period of time, where the pages compose a logical area. Because of the isolation of areas, updates on different areas can be performed concurrently. To allow the areas to be fairly well used, each process is assigned with a random priority for thrashing control. The thrashing control mechanism is implemented on a Linux-based DSM system. Performance results show that the execution time of the applications that are apt to cause system thrashing can be significantly reduced by our approach.

Scaling out NUMA-Aware Applications with RDMA-Based Distributed Shared Memory

The multicore evolution has stimulated renewed interests in scaling up applications on shared-memory multiprocessors, significantly improving the scalability of many applications. But the scalability is limited within a single node; therefore programmers still have to redesign applications to scale out over multiple nodes. This paper revisits the design and implementation of distributed shared memory (DSM) as a way to scale out applications optimized for non-uniform memory access (NUMA) architecture over a well-connected cluster. This paper presents MAGI, an efficient DSM system that provides a transparent shared address space with scalable performance on a cluster with fast network interfaces. MAGI is unique in that it presents a NUMA abstraction to fully harness the multicore resources in each node through hierarchical synchronization and memory management. MAGI also exploits the memory access patterns of big-data applications and leverages a set of optimizations for remote direct memory access (RDMA) to reduce the number of page faults and the cost of the coherence protocol. MAGI has been implemented as a user-space library with pthread-compatible interfaces and can run existing multithreaded applications with minimized modifications. We deployed MAGI over an 8-node RDMAenabled cluster. Experimental evaluation shows that MAGI achieves up to 9.25x speedup compared with an unoptimized implementation, leading to a scalable performance for large-scale data-intensive applications.

Design and Cost Analysis for a Fault-Tolerant Distributed Shared Memory System

Algorithms implementing distributed shared memory (DSM) were developed for ensuring consistency. The performance of DSM algorithms is dependent on system and usage parameters. However, ensuring these algorithms to tolerate faults is a problem that needs to be researched. In this study, we proposed fault-tolerant scheme for DSM system and analyzed reliability and fault-tolerant overhead. Using our analysis, we can choose a proper algorithm for DSM on error prone environment.

Delay tolerant lazy release consistency for distributed shared memory in opportunistic networks

Opportunistic networks (ONs) allow mobile wireless devices to interact with one another through a series of opportunistic contacts. While ONs exploit mobility of devices to route messages and distribute information, the intermittent connections among devices make many traditional computer collaboration paradigms, such as distributed shared memory (DSM), very difficult to realize. DSM systems, developed for traditional networks, rely on relatively stable, consistent connections among participating nodes to function properly.We propose a novel delay tolerant lazy release consistency (DTLRC) mechanism for implementing distributed shared memory in opportunistic networks. DTLRC permits mobile devices to remain independently productive while separated, and provides a mechanism for nodes to regain coherence of shared memory if and when they meet again. DTLRC allows applications to utilize the most coherent data available, even in the challenged environments typical to opportunistic networks. Simulations demonstrate that DTLRC is a viable concept for enhancing cooperation among mobile wireless devices in opportunistic networking environment.

A load balanced directory for distributed shared memory objects

We present MultiBend, a novel distributed directory protocol for shared objects, suitable for large-scale distributed shared memory systems that use d-dimensional mesh-based topologies, where d≥2. Each shared object has an owner node that can modify its value. The ownership may change by moving the object from one node to another in response to move requests. The value of an object can be read by other nodes with lookup requests. MultiBend balances the load of the network edges and nodes by forwarding each move or lookup request and response along a path consisting of multiple bends in the mesh. Using an oblivious routing protocol, the multi-bend paths have a small number of overlaps which helps to reduce the maximum edge and node utilization to achieve load balancing. At the same time, MultiBend achieves small stretch for the total path length of any sequence of move requests, compared to the total optimal path length. MultiBend guarantees O(d2logn) approximation for the load, and O(dlogn) approximation for the stretch due to move requests, where n is the number of nodes in the mesh network. It also guarantees O(d2) approximation for the stretch of lookup requests. We evaluate MultiBend with simulations using various sequences of move and lookup operations in a 16×16 nodes 2-dimensional mesh network. We compare the simulation results to other protocols which are not tailored for load balancing and we find that our protocol is better by as much as the factor of 6.85 in terms of congestion in the worst-case. To the best of our knowledge, this is the first distributed shared memory directory protocol that considers the network load balancing aspect and achieves good approximation ratio for both the load and the stretch.

DVM: A Big Virtual Machine for Cloud Computing

As cloud-based computation grows to be an increasingly important paradigm, providing a general computational interface to support datacenter-scale programming has become an imperative research agenda. Many cloud systems use existing virtual machine monitor (VMM) technologies, such as Xen, VMware, and Windows Hypervisor, to multiplex a physical host into multiple virtual hosts and isolate computation on the shared cluster platform. However, traditional multiplexing VMMs do not scale beyond one single physical host, and it alone cannot provide the programming interface and cluster-wide computation that a datacenter system requires. We design a new instruction set architecture, DISA, to unify myriads of compute nodes to form a big virtual machine called DVM and present programmers the view of a single computer, where thousands of tasks run concurrently in a large, unified, and snapshotted memory space. The DVM provides a simple yet scalable programming model and mitigates the scalability bottleneck of traditional distributed shared memory systems. Along with an efficient execution engine, the capacity of a DVM can scale up to support large clusters. We have implemented and tested DVM on four platforms, and our evaluation shows that DVM has excellent performance and scalability. On one physical host, the system overhead of DVM is comparable to that of traditional VMMs. On 16 physical hosts, the DVM runs 10 times faster than MapReduce/Hadoop and X10. On 160 compute nodes in the TH-1/GZ supercomputer, the DVM delivers a $\bf{12.99\times}$ speedup over the computation on 10 compute nodes. The implementation of DVM also allows it to run above traditional VMMs, and we verify that DVM shows linear speedup on a parallelizable workload on 256 large EC2 instances.

On the power of breakable objects

In distributed shared-memory systems, a breakable object is one that may enter a special broken state, after which all operations will fail and return a special broken symbol B. Most breakable obje...

On the power of breakable objects

In distributed shared-memory systems, a breakable object is one that may enter a special broken state, after which all operations will fail and return a special broken symbol ⊥B. Most breakable objects appeared in literature belong to a class we call acyclic breakable objects, in which every operation moves the object closer to the broken state. We show in general that acyclic breakable objects have limited consensus power. Our main results focus on the richer but less covered cyclic breakable objects. We study the consensus numbers of cyclic breakable objects by using a general algorithmic framework, and obtain several interesting results. In particular one result on a breakable queue points out a mistake that appeared both in a previous paper and a textbook. We further study operation-wise breakable objects in which some operations on the object may break but not other operations. We use various breakable queues as running examples, and demonstrate that the generic framework can be applied in this context as well, and in some cases with nontrivial implementations of the framework. We also provide matching impossibility results to obtain exact consensus numbers for all the cases we consider. Finally, since all example objects we found in the literature with a generic consensus number n are breakable objects, we provide a simple and natural non-breakable object class with generic consensus number n.

DVM

As cloud-based computation becomes increasingly important, providing a general computational interface to support datacenter-scale programming has become an imperative research agenda. Many cloud systems use existing virtual machine monitor (VMM) technologies, such as Xen, VMware, and Windows Hypervisor, to multiplex a physical host into multiple virtual hosts and isolate computation on the shared cluster platform. However, traditional multiplexing VMMs do not scale beyond one single physical host, and it alone cannot provide the programming interface and cluster-wide computation that a datacenter system requires. We design a new instruction set architecture, DISA, to unify myriads of compute nodes to form a big virtual machine called DVM, and present programmers the view of a single computer where thousands of tasks run concurrently in a large, unified, and snapshotted memory space. The DVM provides a simple yet scalable programming model and mitigates the scalability bottleneck of traditional distributed shared memory systems. Along with an efficient execution engine, the capacity of a DVM can scale up to support large clusters. We have implemented and tested DVM on three platforms, and our evaluation shows that DVM has excellent performance in terms of execution time and speedup. On one physical host, the system overhead of DVM is comparable to that of traditional VMMs. On 16 physical hosts, the DVM runs 10 times faster than MapReduce/Hadoop and X10. On 256 EC2 instances, DVM shows linear speedup on a parallelizable workload.

Data race avoidance and replay scheme for developing and debugging parallel programs on distributed shared memory systems

Distributed shared memory (DSM) allows parallel programs to run on distributed computers by simulating a global virtual shared memory, but data racing bugs may easily occur when the threads of a multi-threaded process concurrently access the physically distributed memory. Earlier tools to help programmers locate data racing bugs in non-DSM parallel programs are not easily applied to DSM systems. This study presents the data race avoidance and replay scheme (DRARS) to assist debugging parallel programs on DSM or multi-core systems. DRARS is a novel tool which controls the consistency protocol of the target program, automatically preventing a large class of data racing bugs when the parallel program is subsequently run, obviating much of the need for manual debugging. For data racing bugs that cannot be avoided automatically, DRARS performs a deterministic replay-type function on DSM systems, faithfully reproducing the behavior of the parallel program during run time. Because one class of data racing bugs has already been eliminated, the remaining manual debugging task is greatly simplified. Unlike previous debugging methods, DRARS does not require that the parallel program be written in a specific style or programming language. Moreover, DRARS can be implemented in most consistency protocols. In this paper, DRARS is realized and verified in real experiments using the eager release consistency protocol on a DSM system with various applications.

An asymmetric distributed shared memory model for heterogeneous parallel systems

Heterogeneous computing combines general purpose CPUs with accelerators to efficiently execute both sequential control-intensive and data-parallel phases of applications. Existing programming models for heterogeneous computing rely on programmers to explicitly manage data transfers between the CPU system memory and accelerator memory. This paper presents a new programming model for heterogeneous computing, called Asymmetric Distributed Shared Memory (ADSM), that maintains a shared logical memory space for CPUs to access objects in the accelerator physical memory but not vice versa. The asymmetry allows light-weight implementations that avoid common pitfalls of symmetrical distributed shared memory systems. ADSM allows programmers to assign data objects to performance critical methods. When a method is selected for accelerator execution, its associated data objects are allocated within the shared logical memory space, which is hosted in the accelerator physical memory and transparently accessible by the methods executed on CPUs. We argue that ADSM reduces programming efforts for heterogeneous computing systems and enhances application portability. We present a software implementation of ADSM, called GMAC, on top of CUDA in a GNU/Linux environment. We show that applications written in ADSM and running on top of GMAC achieve performance comparable to their counterparts using programmer-managed data transfers. This paper presents the GMAC system and evaluates different design choices. We further suggest additional architectural support that will likely allow GMAC to achieve higher application performance than the current CUDA model.

An asymmetric distributed shared memory model for heterogeneous parallel systems

Heterogeneous computing combines general purpose CPUs with accelerators to efficiently execute both sequential control-intensive and data-parallel phases of applications. Existing programming models for heterogeneous computing rely on programmers to explicitly manage data transfers between the CPU system memory and accelerator memory. This paper presents a new programming model for heterogeneous computing, called Asymmetric Distributed Shared Memory (ADSM), that maintains a shared logical memory space for CPUs to access objects in the accelerator physical memory but not vice versa. The asymmetry allows light-weight implementations that avoid common pitfalls of symmetrical distributed shared memory systems. ADSM allows programmers to assign data objects to performance critical methods. When a method is selected for accelerator execution, its associated data objects are allocated within the shared logical memory space, which is hosted in the accelerator physical memory and transparently accessible by the methods executed on CPUs. We argue that ADSM reduces programming efforts for heterogeneous computing systems and enhances application portability. We present a software implementation of ADSM, called GMAC, on top of CUDA in a GNU/Linux environment. We show that applications written in ADSM and running on top of GMAC achieve performance comparable to their counterparts using programmer-managed data transfers. This paper presents the GMAC system and evaluates different design choices. We further suggest additional architectural support that will likely allow GMAC to achieve higher application performance than the current CUDA model.

OPDSM: A Combinatorial Object-Based and Page-Based DSM Model

Distributed Shared Memory (DSM) system is designed on the basis of page-based, shared-variable-based or object-based access. There are certain advantages and disadvantages of each access methodology. Comparison of such different access types in a distributed environment is based on parameters namely, type of operation (data read or update), maintenance of consistent data copies on all nodes accessing shared data, administration of data manipulations on multi-computers, etc. This paper compares the design and implementation issues for page-based and object-based DSM in a multicomputing environment. The paper also proposes a hybrid model for implementation of a DSM system that combines some features of page-based and object-based access that is focused on efficient data access with optimum performance with respect to memory access and cohesive data/object organization. Thus the hybrid model is proposed to get a common function of data sharing by using the best features of both page-based and object- based DSM models.

Scalability of relaxed consistency models in NoC based multicore architectures

This paper studies realization of relaxed memory consistency models in the network-on-chip based distributed shared memory (DSM) multi-core systems. Within DSM systems, memory consistency is a critical issue since it affects not only the performance but also the correctness of programs. We investigate the scalability of the relaxed consistency models (weak, release consistency) implemented by using transaction counters. Our experimental results compare the average and maximum code, synchronization and data latencies of the two consistency models for various network sizes with regular mesh topologies. The observed latencies rise for both the consistency models as the network size grows. However, the scaling behaviors are different. With the release consistency model these latencies grow significantly slower than with the weak consistency due to better optimization potential by means of overlapping, reordering and program order relaxations. The release consistency improves the performance by 15.6% and 26.5% on average in the code and consistency latencies over the weak consistency model for the specific application, as the system grows from single core to 64 cores. The latency of data transactions grows 2.2 times faster on the average with a weak consistency model than with a release consistency model when the system scales from single core to 64 core

Specification-based Verification in a Distributed Shared Memory Simulation Model

The emergence of chip multiprocessors is leading to rapid advances in hardware and software systems to provide distributed shared memory (DSM) programming models, so-called DSM systems. A DSM system provides programming advantages within a scalable and cost-effective hardware solution. This benefit derives from the fact that a DSM system creates a shared-memory abstraction on top of a distributed-memory machine by caching data replicas locally. In this respect, a coherence protocol is a vital component responsible for assuring data consistency across all replicas. The design of coherence protocols impacts a DSM system in terms of both performance and accuracy. Performance is often measured via simulation and various verification techniques have been proposed to deal with protocol accuracy. Nevertheless, integrating accuracy verification into a DSM cluster simulation to ensure correct simulation results is still an open issue. In this paper, we address three properties of a coherence protocol (safety, liveness, and inclusion) without which errors may occur in the simulation results. We propose a specification-based parameter—model interaction (SPMI) technique to detect these cases in a particular DSM cluster simulator called DSiMCluster. Our experimental results demonstrate that with SPMI, DSiMCluster can ensure the coherence protocol properties and provides a correct reflection of memory characteristics in shared-memory and DSM multiprocessors.

A meta‐predictor framework for prefetching in object‐based DSMs

AbstractDynamic optimizers modify the binary code of programs at runtime by profiling and optimizing certain aspects of the execution. We present a completely software‐based framework that dynamically optimizes programs for object‐based distributed shared memory (DSM) systems on clusters. In DSM systems, reducing the number of messages between cluster nodes is crucial. Prefetching transfers data in advance from the storage node to the local node so that communication is minimized. Our framework uses a profiler and a dynamic binary rewriter that monitor the access behavior of the application and place prefetches where they are beneficial to speed up the application. In addition, we use two distinct predictors to handle different types of access patterns. A meta‐predictor analyzes the memory access behavior and dynamically enables one of the predictors. Our system also adapts the number of prefetches per request to best fit the application's behavior. The evaluation shows that the performance of our system is better than the manual prefetching. The number of messages sent decreases by up to 90%. Performance gains of up to 80% can be observed on benchmarks. Copyright © 2009 John Wiley & Sons, Ltd.

DSiMCluster: A Simulation Model for Efficient Memory Analysis Experiments of DSM Clusters

The emergence of chip multiprocessors is leading to the rapid development of multicore clusters that provide distributed—shared memory (DSM) programming models, so called DSM systems. DSM systems allow applications to access data in a logically shared address space by abstracting away the distinction of physical memory location. However, a significant drawback of a DSM cluster is that it performs poorly on applications that require large shared memory because of the performance loss or overhead caused by memory locality. Consequently, an effective memory analysis methodology is mandatory to quantify the impact of such overhead. In this respect, a widely used technique is simulation. Nevertheless, both the intrinsically difficult design of cluster architectures and the rapid advance of DSM implementations impose a constraint on simulation model designs. In this paper we describe DSiMCluster, a simulation model for memory analysis experiments of a DSM cluster system. We capitalize on the observation that the components of a simulation model have to be highly reusable and provide automatic behavioral verification. That is, in order to maintain the accuracy of simulation results with the technological advances of target architectures, which grow extensively at the software level. We show that highly reusable simulation components can be created by implementing a system-oriented simulation on top of a well-founded and broadly applicable, discrete-event simulation engine. Our experimental results demonstrate that, with these operations, DSiMCluster provides an efficient framework for memory analysis experiments on a DSM cluster architecture within a reconfigurable, customizable and user-friendly environment.

Special Issue: Euro‐Par 2008

AbstractPreface for the special issue on Euro‐Par 2008. Copyright © 2009 John Wiley & Sons, Ltd.