Optimistic Replication Research Articles

Approaches to Conflict-free Replicated Data Types

Conflict-free Replicated Data Types (CRDTs) allow optimistic replication in a principled way. Different replicas can proceed independently, being available even under network partitions, and always converging deterministically: replicas that have received the same updates will have equivalent state, even if received in different orders. After a historical tour of the evolution from sequential data types to CRDTs, we present in detail the two main approaches to CRDTs, operation-based and state-based, including two important variations, the pure operation-based and the delta-state based. Intended for prospective CRDT researchers and designers, this paper provides solid coverage of the essential concepts, clarifying some misconceptions which frequently occur, but also presents some novel insights gained from considerable experience in designing both specific CRDTs and approaches to CRDTs.

Supporting conflict-free replicated data types in opportunistic networks

Conflict-Free Replicated Data Types (CRDTs) are data types that can be used in distributed systems when optimistic replication is tolerable. Replicas can be updated locally, without coordination, and consistency is obtained eventually by asynchronously propagating updates among replicas. Because CRDTs can tolerate asynchronous transmissions, they can serve as software elements in opportunistic networks (OppNets), where the dissemination of information is dependent on unplanned transient radio contacts between mobile nodes. In this paper we investigate the problem of implementing operation-based, state-based, and delta-state-based CRDTs in OppNets. A contact-driven synchronization algorithm is proposed for each kind of CRDT, and experiments based on realistic tracesets are conducted in order to compare how these algorithms can perform in an OppNet. Experimental results show that delta-state-based CRDTs globally outperforms operation-based and pure state-based CRDTs, especially when considering the number of messages required to ensure the synchronization of replicas.

SconeKV: A Scalable, Strongly Consistent Key-Value Store

For decades, relational databases provided a strong foundation for constructing applications due to their ACID properties. However, distributed applications reached a scale, both in terms of data volume and number of concurrent clients, that traditional databases cannot accommodate. NoSQL databases addressed this problem by trading consistency for scalability, namely through horizontal scalability schemes supported by optimistic replication protocols, which only guarantee eventual consistency. In this paper, we explore a novel design between the two extremes, which is able to scale to large deployments while still offering strong consistency guarantees in the form of serializable transactions. Our key insight is to leverage recent advances in membership services that provide strongly consistent views at scale. Those assurances from the membership layer simplify building efficient and consistent storage protocols. Our evaluation of the resulting system, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SconeKV</small> , in a realistic scenario shows that it scales and performs better than CockroachDB while being competitive with Cassandra.

A Constraint-based Approach for Generating Transformation Patterns

Undoing operations is an indispensable feature for many collaborative applications, mainly collaborative editors. It provides the ability to restore a correct state of shared data after erroneous operations. In particular, selective undo allows to undo any operation and is based on rearranging operations in the history thanks to the Operational Transformation (OT) approach. OT is an optimistic replication technique allowing for updating the shared data concurrently while maintaining convergence. It is a challenging task how to meaningfully combine OT and undo approaches. Indeed, undoing operations that are received and executed out-of-order at different sites leads to divergence cases. Even though various undo solutions have been proposed over the recent years, they are either limited or erroneous. In this paper, we propose a constraint-based approach to address the undo problem. We use Constraint Satisfaction Problem (CSP) theory to devise correct and undoable transformation patterns (w.r.t OT and undo properties) which considerably simplifies the design of collaborative objects.

Verifying eventual consistency of optimistic replication systems

We address the verification problem of eventual consistency of optimistic replication systems. Such systems are typically used to implement distributed data structures over large scale networks. We introduce a formal definition of eventual consistency that applies to a wide class of existing implementations, including the ones using speculative executions. Then, we reduce the problem of checking eventual consistency to reachability and model checking problems. This reduction enables the use of existing verification tools for message-passing programs in the context of verifying optimistic replication systems. Furthermore, we derive from these reductions decision procedures for checking eventual consistency of systems implemented as finite-state programs communicating through unbounded unordered channels.

A Fault-Tolerant Mobile Computing Model Based On Scalable Replica

The most frequent challenge faced by mobile user is stay connected with online data, while disconnected or poorly connected store the replica of critical data. Nomadic users require replication to store copies of critical data on their mobile machines. Existing replication services do not provide all classes of mobile users with the capabilities they require, which include: the ability for direct synchronization between any two replicas, support for large numbers of replicas, and detailed control over what files reside on their local (mobile) replica. Existing peer-to- peer solutions would enable direct communication, but suffers from dramatic scaling problems in the number of replicas, limiting the number of overall users and impacting performance. Roam is a replication system designed to satisfy the requirements of the mobile user. Roam is based on the Ward Model, replication architecture for mobile environments. Using the Ward Model and new distributed algorithms, Roam provides a scalable replication solution for the mobile user. We describe the motivation, design, and implementation of Roam and report its performance. Replication is extremely important in mobile environments because nomadic users require local copies of important data. Model, a new replication architecture that has been designed especially with mobility in mind, though of course it applies equally well to stationary environments (1). The Ward Model provides a new replication paradigm that is neither strictly peer nor client-server; rather, it is a hybrid model of the two that allows everyone to be effectively a peer while maintaining good scalability in the number of replicas. Additionally, Roam provides a number of new distributed algorithms. For instance, Roam contains new and improved distributed algorithms for garbage collection and for dynamic management of the version vector, the main data structure behind most optimistic replication systems. In this paper we present an approach for data replication that considers the mobility of clients: The data sets which have to be replicated to mobile clients depend often on dynamic parameters like location of the client or time. Therefore, we introduce a generic model for specifying fragments of the global database with respect to such parameters. The increasing availability of mobile devices and wireless

An Adaptive-Grained Consistency Maintenance Scheme for Shared Data on Emergency and Rescue Applications

Efficient information sharing is difficult to achieve in the scenario of emergency and rescue operations because there is no communication infrastructure at the disaster sites. In general, the network condition is relatively reliable in the intra-site environment but relatively unreliable in the inter-site environment. The network partitioning problem may occur between two sites. Although one can exploit the replication technique used in data grid to improve the information availability in emergency and rescue applications, the data consistency problem occurs between replicas. In this paper, the authors propose a middleware called “Seagull” to transparently manage the data availability and consistency issues of emergency and rescue applications. Seagull adopts the optimistic replication scheme to provide the higher data availability in the inter-site environment. It also adopts the pessimistic replication scheme to provide the stronger data consistency guarantee in the intra-site environment. Moreover, it adopts an adaptive consistency granularity strategy that achieves the better performance of the consistency management because this strategy provides the higher parallelism when the false sharing happens. Lastly, Seagull adopts the transparency data consistency management scheme, and thus the users do not need to modify their source codes to run on the Seagull.

Design Thinking and Cloud Manufacturing: A Study of Cloud Model Sharing Platform Based on Separated Data Log

To solve the product data consistency problem which is caused by the portable system that cannot conduct real-time update of product data in mobile environment under the mass customization production mode, a new product data optimistic replication method based on log is presented. This paper focuses on the design thinking provider, probing into a manufacturing resource design thinking cloud platform based on manufacturing resource-locating technologies, and also discuss several application scenarios of cloud locating technologies in the manufacturing environment. The actual demand of manufacturing creates a new mode which is service-oriented and has high efficiency and low consumption. Finally, they are different from the crowd-sourcing application model of Local-Motors. The sharing platform operator is responsible for a master plan for the platform, proposing a open interface standard and establishing a service operation mode.

FBSA: A Flash Memory-based Scheduling Algorithm for Optimistic Replication

FBSA: A Flash Memory-based Scheduling Algorithm for Optimistic Replication

Evaluation of intra‐group optimistic data replication in P2P groupware systems

SummaryPeer‐to‐peer (P2P) computing systems have become very popular during the last years because of their ability to scale to a large number of users and efficient communication among peers. They can support complex computational processes, beyond simple file sharing, while offering advantages of decentralized distributed systems. However, such systems may suffer from availability and reliability. To increase availability and reliability, and therefore, improve the perception of peers, yielding to fast response times and rich experience, data replication techniques are the foremost means in such systems. Indeed, in many P2P applications, for example, in a groupware, documents generated along application life cycle can change over time. The need is then to efficiently replicate dynamic documents and data to support group processes and collaboration. In this paper, we propose a replication system for documents structured as XML files and evaluate it under different scenarios. The proposed system has a super‐peer architecture that provides fast consistency for late joining peers. It uses optimistic replication techniques with propagating update operations from source to destination node in push mode. The system is suitable for asynchronous collaboration in online collaborative teams accomplishing a common project in a P2P environment. Copyright © 2012 John Wiley &amp; Sons, Ltd.

Replication Management and Optimistic Replication Challenges in Mobile Environment

Wireless and mobile communication is an important technology that improves user’s daily life and facilitates the development of some new technology such as electronic commerce and mobile commerce. Replication is a fundamental technique for supporting concurrent work practices in mobile environments. Optimistic replication algorithms allow replica contents to become stale but in a controlled way. In return, they become far more efficient and available than traditional replication algorithms that keep all the replicas consistent, especially when the network and computers are unreliable. The use of optimistic replication has grown explosively due to the proliferation of the use of the Internet and mobile computing devices, but its systematic study has begun only recently. This paper explores the various replication strategies, replication in mobile environment, benefits and challenges of optimistic replication in mobile applications.

Normalization of edit sequences for text synchronization

It often occurs that local copies of a text are modified by users but that the local modifications are not synchronized (thus allowing the merged text to become the source for later editions) until later when, for instance the network connection is reestablished. Since text editions usually affect a small fraction of the whole content, the history of edit operations provides a compact representation of the modified file. In this paper, we define a normal form for these records which will permit for the comparison of all text files that have been obtained by editing a common source S when the difference between each output file Oi and the source file is given as a sequence Li of edit operations. We show that the normalized sequence is unique for all the equivalent text editions and provide efficient procedures with which to compute this normal form and to obtain the edit sequence LM transforming S into a merged file M which integrates all the local modifications. We also discuss how these normalization can be integrated into the operational transformation paradigm for optimistic replication.

Replicated abstract data types: Building blocks for collaborative applications

For distributed applications requiring collaboration, responsive and transparent interactivity is highly desired. Though such interactivity can be achieved with optimistic replication, maintaining replica consistency is difficult. To support efficient implementations of collaborative applications, this paper extends a few representative abstract data types (ADTs), such as arrays, hash tables, and growable arrays (or linked lists), into replicated abstract data types (RADTs). In RADTs, a shared ADT is replicated and modified with optimistic operations. Operation commutativity and precedence transitivity are two principles enabling RADTs to maintain consistency despite different execution orders. Especially, replicated growable arrays (RGAs) support insertion/deletion/update operations. Over previous approaches to the optimistic insertion and deletion, RGAs show significant improvement in performance, scalability, and reliability.

A New Optimistic Replication Strategy For Large-Scale Mobile Distributed Database Systems

This paper proposes a new optimistic replication strategy for maintaining the eventual consistency in large-scale mobile distributed database systems. The proposed strategy consists of three components in order to support the characteristics of such systems. These components are: replication architecture, updates propagation protocol, and replication method. The purpose of the replication architecture is to provide a comprehensive infrastructure for distributing replicas among wide areas. The purpose of the propagation protocol is to transfer data updates between the components of the replication architecture in a manner that achieves the eventual consistency of data and improves the availability of recent updates to all replicas. The replication method provides a mechanism for implementing the propagation protocol, and automating the propagation of updates among the different replicas. The effectiveness of the proposed strategy is compared with two baseline replication strategies and shown that it achieves updates propagation delay reduction. Also, the results showed that the horizontal extension provided by the proposed strategy is more suitable than the vertical extension for large scale mobile distributed database systems.

On Coordinating Collaborative Objects

A collaborative object represents a data type (such as a text document) designed to be shared by a group of dispersed users. The Operational Transformation (OT) is a coordination approach used for supporting optimistic replication for these objects. It allows the users to concurrently update the shared data and exchange their updates in any order since the convergence of all replicas, i.e. the fact that all users view the same data, is ensured in all cases. However, designing algorithms for achieving convergence with the OT approach is a critical and challenging issue. In this paper, we propose a formal compositional method for specifying complex collaborative objects. The most important feature of our method is that designing an OT algorithm for the composed collaborative object can be done by reusing the OT algorithms of component collaborative objects. By using our method, we can start from correct small collaborative objects which are relatively easy to handle and incrementally combine them to build more complex collaborative objects.

Building a collaborative peer-to-peer wiki system on a structured overlay

The ever growing request for digital information raises the need for content distribution architectures providing high storage capacity, data availability and good performance. While many simple solutions for scalable distribution of quasi-static content exist, there are still no approaches that can ensure both scalability and consistency for the case of highly dynamic content, such as the data managed inside wikis. We propose a peer-to-peer solution for distributing and managing dynamic content, that combines two widely studied technologies: Distributed Hash Tables (DHT) and optimistic replication. In our ''universal wiki'' engine architecture (UniWiki), on top of a reliable, inexpensive and consistent DHT-based storage, any number of front-ends can be added, ensuring both read and write scalability, as well as suitability for large-scale scenarios. The implementation is based on Damon, a distributed AOP middleware, thus separating distribution, replication, and consistency responsibilities, and also making our system transparently usable by third party wiki engines. Finally, UniWiki has been proved viable and fairly efficient in large-scale scenarios.

Scalable and topology-aware reconciliation on P2P networks

Collaborative applications are characterized by high levels of data sharing. Optimistic replication has been suggested as a mechanism to enable highly concurrent access to the shared data, whilst p...

Timed buffers: A technique for update propagation in nomadic environments

Optimistic replication algorithms allow data presented to users to be stale (non-up-to-date) but in a controlled way: they propagate updates in background and allow any replica to be accessed directly most of the time. When the timely propagation of updates to remote distributed replicas is an important issue, it is preferable that a replica gets the same update twice than it does not receive it at all. On the other hand, few assumptions on the topology of the network can be made in a nomadic environment, where connections are likely to change unpredictably. An extreme approach would be to blindly “push” every update to every replica; however, this would lead to a huge waste of bandwidth and of resources. In this paper, we present a novel approach based on timed buffers, a technique that tends to reduce the overall number of propagated updates while guaranteeing that every update is delivered to every replica and that the propagation is not delayed.

Concise version vectors in WinFS

Conflicts naturally arise in optimistically replicated systems. The common way to detect update conflicts is via version vectors, whose storage and communication overhead are number of replicas × number of objects. These costs may be prohibitive for large systems. This paper presents predecessor vectors with exceptions (PVEs), a novel optimistic replication technique developed for Microsoft’s WinFS system. The paper contains a systematic study of PVE’s performance gains over traditional schemes. The results demonstrate a dramatic reduction of storage and communication overhead in normal scenarios, during which communication disruptions are infrequent. Moreover, they identify a cross-over threshold in communication failure-rate, beyond which PVEs loses efficiency compared with traditional schemes.

Using Permuted States and Validated Simulation to Analyze Conflict Rates in Optimistic Replication

Optimistic replication provides high data availability in the presence of network outages. Although widely deployed, this relaXed consistency model introduces concurrent updates, whose behavior is poorly understood due to the vast state space. This paper introduces the notion of permuted states to eliminate system states that are redundant and unreachable, which can constitute the majority of states (4069 out of 4096 for four replicas). With the aid of permuted states, we are for the first time able to construct analytical models beyond the two-replica case. By eXamining the analysis for 2 to 4 replicas, we can demystify the process of forming identical conflicts—the most common conflict type at high replication factors. Additionally, we have automated and optimized the generation of permuted states, which allows us to eXplore higher replication factors (up to 10 replicas) using hybrid techniques. It also allows us to validate our results with eXisting simulations based on actual replication mechanisms, which previously were analytically validated with only one pair of replicas. Finally, we have discovered that update locality and bimodal access patterns are the primary factors contributing to the formation of identical conflicts.