Commutative Replicated Data Type Research Articles

Real Differences between OT and CRDT in Correctness and Complexity for Consistency Maintenance in Co-Editors

OT (Operational Transformation) was invented for supporting real-time co-editors in the late 1980s and has evolved to become core techniques widely used in today's working co-editors and adopted in industrial products. CRDT (Commutative Replicated Data Type) for co-editors was first proposed around 2006, under the name of WOOT (WithOut Operational Transformation). Follow-up CRDT variations are commonly labeled as "post-OT" techniques capable of making concurrent operations natively commutative in co-editors. On top of that, CRDT solutions have made broad claims of superiority over OT solutions, and often portrayed OT as an incorrect and inefficient technique. Over one decade later, however, CRDT is rarely found in working co-editors; OT remains the choice for building the vast majority of today's co-editors. Contradictions between the reality and CRDT's purported advantages have been the source of much confusion and debate in co-editing researcher and developer communities. To seek truth from facts, we set out to conduct a comprehensive and critical review on representative OT and CRDT solutions and working co-editors based on them. From this work, we have made important discoveries about OT and CRDT, and revealed facts and evidences that refute CRDT claims over OT on all accounts. These discoveries help explain the underlying reasons for the choice between OT and CRDT in the real world. We report these results in a series of three articles. In this article (the second in the series), we reveal the differences between OT and CRDT in their basic approaches to realizing the same general transformation and how such differences had resulted in different technical challenges and consequential correctness and complexity issues. Moreover, we reveal hidden complexity and algorithmic flaws with representative CRDT solutions, and discuss common myths and facts related to correctness and complexity of OT and CRDT. We hope the discoveries from this work help clear up common myths and confusions surrounding OT and CRDT, and accelerate progress in co-editing technology for real world applications.

Real Differences between OT and CRDT under a General Transformation Framework for Consistency Maintenance in Co-Editors

OT (Operational Transformation) was invented for supporting real-time co-editors in the late 1980s and has evolved to become a collection of core techniques widely used in today's working co-editors and adopted in major industrial products. CRDT (Commutative Replicated Data Type) for co-editors was first proposed around 2006, under the name of WOOT (WithOut Operational Transformation). Follow-up CRDT variations are commonly labeled as "post-OT" techniques capable of making concurrent operations natively commutativity in co-editors. On top of that, CRDT solutions have made broad claims of superiority over OT solutions, and routinely portrayed OT as an incorrect, complex and inefficient technique. Over one decade later, however, CRDT is rarely found in working co-editors, and OT remains the choice for building the vast majority of today's co-editors. Contradictions between the reality and CRDT's purported advantages have been the source of much confusion and debate in co-editing research and developer communities. Have the vast majority of co-editors been unfortunate in choosing the faulty and inferior OT, or those CRDT claims are false? What are the real differences between OT and CRDT for co-editors? What are the key factors and underlying reasons behind the choices between OT and CRDT in the real world? A thorough examination of these questions is relevant not only to researchers who are exploring the frontiers of co-editing technologies and systems, but also to practitioners who are seeking viable techniques to build real world applications. To seek truth from facts, we set out to conduct a comprehensive and critical review on representative OT and CRDT solutions and working co-editors based on them. From this work, we have made important discoveries about OT and CRDT, and revealed facts and evidences that refute CRDT claims over OT on all accounts. We report our discoveries in a series of articles and the current article is the first one in this series. In this paper, we present a general transformation framework for consistency maintenance in co-editors, which was distilled from dissecting and examining representative OT and CRDT solutions (and other alternative solutions) during this work, and report our discoveries under the guidance of this framework. In particular, we reveal that CRDT is like OT in following a general transformation approach, but achieves the same transformation indirectly, in contrast to OT direct transformation approach; and CRDT is not natively commutative for concurrent co-editing operations, but has to achieve the same OT commutativity indirectly as well, with consequential correctness and complexity issues. Uncovering the hidden transformation nature and demystifying the commutativity property of CRDT provides much-needed clarity about what CRDT really is and is not to co-editing, and serves as the foundation to explore the real differences between OT and CRDT in correctness, complexity, implementation, and real world applications, which are reported in follow-up articles. We hope discoveries from this work help clear up common misconceptions and confusions surrounding OT and CRDT, and accelerate progress in co-editing technology for real world applications.?

Supporting selective undo of string-wise operations for collaborative editing systems

With the advancement of distributed computing systems, an increasing number of collaborative systems are developed to support real-life collaborative work. As a typical representative of collaborative systems, collaborative editing systems have been one of the major researches within CSCW(Computer Supported Cooperative Work) areas. In collaborative editing systems, selective undo is a key feature that permits users to undo any operation at anywhere and anytime. Recently, with the extensive popularity of big data and cloud computing, collaborative editing systems increasingly tend to massive-scale collaborations, which have to face new technical challenges. How to support selective undo of string-wise operations has become one of significant challenges in future generation collaborative editing systems. Although CRDT (Commutative Replicated Data Type) is a new generation of collaborative editing algorithms, most CRDT algorithms do not support selective undo of string-wise operations. Currently, existing works take time logarithmic or even linear in the number of operation nodes to undo an operation, which would be suboptimal in which a large amount of operations may accumulate. In addition, how to formally prove the correctness of the algorithm is necessary. This paper proposes an efficient CRDT algorithm that supports integrated do and selective undo of string-wise operations. Formal proofs are provided to prove the correctness of the proposed approach and the intentions preserving of do/undo operations. Theoretical analysis and experimental evaluation show that our algorithm outperforms the existing CRDT algorithms to support selective undo.

Total order in opportunistic networks

SummaryOpportunistic network applications are usually assumed to work only with unordered immutable messages, like photos, videos, or music files, while applications that depend on ordered or mutable messages, like chat or shared contents editing applications, are ignored. In this paper, we examine how total ordering can be achieved in an opportunistic network. By leveraging on existing dissemination and causal order algorithms, we propose a commutative replicated data type algorithm on the basis of Logoot for achieving total order without using tombstones in opportunistic networks where message delivery is not guaranteed by the routing layer. Our algorithm is designed to use the nature of the opportunistic network to reduce the metadata size compared to the original Logoot, and even to achieve in some cases higher hit rates compared to the dissemination algorithms when no order is enforced. Finally, we present the results of the experiments for the new algorithm by using an opportunistic network emulator, mobility traces, and Wikipedia pages.

A string-wise CRDT algorithm for smart and large-scale collaborative editing systems

With the development of big data and cloud computing, real-time collaborative editing systems have to face new challenges. How to support string-wise operations for smart and large-scale collaborations is one of the key issues in next generation of collaborative editing systems, which is both the core topic of collaborative computing area and the fundamental research of many collaborative systems in science and engineering. However, string-wise operations have troubled the existing collaborative editing algorithms, including Operational Transformation (OT) and Commutative Replicated Data Type (CRDT), for many years. This paper proposes a novel and efficient CRDT algorithm that integrates string-wise operations for smart and massive-scale collaborations. Firstly, the proposed algorithm ensures the convergence and maintains operation intentions of collaborative users under an integrated string-wise framework. Secondly, formal proofs are provided to prove both the correctness of the proposed algorithm and the intentions preserving of string-wise operations. Thirdly, the time complexity of the proposed algorithm has been analyzed in theory to be lower than that of the state of the art OT algorithm and CRDT algorithm. Fourthly, experiment evaluations show that the proposed algorithm outperforms the state of the art OT algorithm and CRDT algorithm.

Meta-operation conflict resolution for human–human interaction in collaborative feature-based CAD systems

Conflicts resolution is one of the key issues in maintaining consistency and in supporting smooth human---human interaction for real-time collaborative systems. This paper presents a novel approach of meta-operation conflict resolution for feature-based collaborative CAD system. Although commutative replicated data type (CRDT) is an emerging technique for conflict resolution, it is not capable of resolving conflicts among meta operations for 3D CAD systems. By defining 3 types of meta operations, this work extends CRDT capability to meta operation conflict resolution from 1D to 3D applications. The paper defines the dependency, casuality, conflict and compatible relations specific for 3D collaborative CAD systems. The conflicts of feature-based operations are automatically detected by tracking topological entity changes with the assistance of a persistent data structure, topological entity structure tree ($$TES\_Tree$$TES_Tree). An efficient commutativity-based confliction combination method is proposed to preserve the design intention of each user in a transparent way and maintains the eventual consistent state of the system. The proposed methods are tested in a prototype system with case studies, time complexity analysis and correctness proof.

Live linked data: synchronising semantic stores with commutative replicated data types

Linked Data is currently interconnecting information on the web, creating the web of data. It allows data consumers to combine different data sets and perform powerful queries. However, this means either to copy data sets locally or perform distributed querying. Local copies have problems of freshness, distributed queries, of scalability and performance. Linked Data producers are currently going live by providing streams of data updates, opening a third way to query: synchronise and search. Each Linked Data node can follow update streams of others, creating a social network of live updates: the Live Linked Data LLD. Unfortunately, synchronising data among autonomous participants raises issues of concurrency and consistency. In this paper, we propose SU-Set, a Commutative Replicated Data Type CRDT for RDF graph updated with SPARQL Update 1.1. We describe how a semantic store can use SU-Set to ensure eventual consistency in LLD, with a low overhead in time, space and communication.

SrCE: a collaborative editing of scalable semantic stores on P2P networks

Commutative Replicated Data Type CRDT is a convergence philosophy invented as a new generation of technique that ensures consistency maintenance of replica in collaborative editors without any difficulty over Peer-to-Peer P2P networks. This technique has been successfully applied to different data representation types in scalable collaborative editing for linear, tree document structure and semi-structured data types but not yet on set data type ensuring Causality, Consistency and Intention CCI preservation criteria. In this paper, we propose a srCE approach, a novel CRDT for a set structure to facilitate the collaborative and concurrent editing of Resource Description Framework RDF stores in large scale by different members of virtual community. Our approach ensures CCI model and is not tied to a specific case and therefore can be applied for any document that complies to set structure. A prototype implementation using Friend of a Friend FOAF data sets with and without the srCE model illustrates significant improvement in scalability and performance.

Consistency without concurrency control in large, dynamic systems

Replicas of a commutative replicated data type (CRDT) eventually converge without any complex concurrency control. We validate the design of a non-trivial CRDT, a replicated sequence, with performance measurements in the context of Wikipedia. Furthermore, we discuss how to eliminate a remaining scalability bottleneck: Whereas garbage collection previously required a system-wide consensus, here we propose a flexible two-tier architecture and a protocol for migrating between tiers. We also discuss how the CRDT concept can be generalised, and its limitations.

Logoot-Undo: Distributed Collaborative Editing System on P2P Networks

Peer-to-peer systems provide scalable content distribution for cheap and resist to censorship attempts. However, P2P networks mainly distribute immutable content and provide poor support for highly dynamic content such as produced by collaborative systems. A new class of algorithms called CRDT (Commutative Replicated Data Type), which ensures consistency of highly dynamic content on P2P networks, is emerging. However, if existing CRDT algorithms support the "edit anywhere, anytime” feature, they do not support the "undo anywhere, anytime” feature. In this paper, we present the Logoot-Undo CRDT algorithm, which integrates the "undo anywhere, anytime” feature. We compare the performance of the proposed algorithm with related algorithms and measure the impact of the undo feature on the global performance of the algorithm. We prove that the cost of the undo feature remains low on a corpus of data extracted from Wikipedia.