Concurrency Control Protocol Research Articles

Dodo: A scalable optimistic deterministic concurrency control protocol

Deterministic concurrency control is able to avoid the expensive two-phase commit in distributed databases and can solve the single-thread bottleneck of transaction processing in blockchain systems. Most existing deterministic concurrency control protocols rely on prior knowledge of the transaction’s read-write set, which is impractical in most cases. The state-of-the-art deterministic concurrency control protocols Aria and DOCC break this limitation. However, they do not perform well in multi-node and multi-core scalability.To solve the scalability issues, we propose Dodo, a novel deterministic concurrency control protocol. Dodo processes transactions in multi-batches, and each batch is divided into three phases. In the first phase, transactions are executed as read committed and staged. In the second phase, transactions are validated for the read-write conflict. In the third phase, only the continuous un-conflicted transactions at the head of the batch can be committed. For the aborted transaction that will be rerun in the next batch, we utilize its write set in the previous execution to reduce the read-write conflicts in the next batch. In this way, Dodo has the following benefits. First, Dodo does not rely on the prior knowledge of the read-write set. Second, Dodo commits transactions in pre-determined orders (TIDs) providing high multi-node scalability. Third, Dodo runs transactions in each phase concurrently, and the aborted transactions are re-executed in a conflict-less manner, enabling high multi-core scalability. Besides, we propose two optimism-based improvements, lazy decision and early-write visibility, to reduce aborts and blocking. Our evaluation shows that Dodo outperforms Aria and DOCC by up to 16.5x and 8.0x in a single-node setting and scales well in a multi-node setting.

EPO‐R: An efficient garbage collection scheme for long‐term transactions

SummaryThis article proposes EPO‐R, which is an efficient garbage collection scheme designed for multi‐version concurrency control (MVCC) protocols. MVCC generates a version for each update operation, and it can exhaust physical memory for dynamically changing environments such as IoT payments. Eager pruning of obsolete versions (EPO) is a novel garbage collection technique for long‐term transactions. We found room for improvement in EPO, which triggers reclamation invocation. EPO is triggered for each write operation, and it wastes CPU resources. The proposed method EPO‐R is triggered on a read operation to address this issue. The result of experiments with 64 CPU cores and workloads with the combination of short‐term and long‐term transactions demonstrated that EPO‐R exhibited 6.3 times higher throughput than that of EPO. The reason for this can be explained by analyzing the number of unreclaimable versions.

Hybrid concurrency control protocol for data sharing among heterogeneous blockchains

Hybrid concurrency control protocol for data sharing among heterogeneous blockchains

Exploiting hashing for concurrent query processing and indexing of current location of moving objects

SummaryDue to recent developments in location‐based services and mobile computing, the need for indices for moving objects has been strengthened to improve the response time of a query operation. With a single index in place for managing both the update and query operations for moving objects, the index needs to be updated each time the object moves. This deteriorates the performance of concurrent query operations. It is critical to handle the conflicts between continuous update and query operations effectively using appropriate concurrency control protocol otherwise, inconsistent results will be reported. Many indices have been proposed in the literature for moving objects but they lack the support for processing concurrent operations. Further, the consistent indices in the literature are based on tree structure that have computationally expensive split/merge operations that can negatively affect the response time of query processing algorithms. Moreover such tree based indices are proposed for historical and future timeline data. As the scope of this article is on current timeline and concurrent continuous query operations, therefore, we exploit state of the art hash‐based indices in the literature and presented the two consistent versions. The comparative analysis of the indices is performed and meaningful findings are also presented along.

Detock: High Performance Multi-region Transactions at Scale

Many globally distributed data stores need to replicate data across large geographic distances. Since synchronously replicating data across such distances is slow, those systems with high consistency requirements often geo-partition data and direct all linearizable requests to the primary region of the accessed data. This significantly improves performance for workloads where most transactions access data close to where they originate from. However, supporting serializable multi-geo-partition transactions is a challenge, and they often degrade the performance of the whole system. This becomes even more challenging when they conflict with single-partition requests, where optimistic protocols lead to high numbers of aborts, and pessimistic protocols lead to high numbers of distributed deadlocks. In this paper, we describe the design of concurrency control and deadlock resolution protocols, built within a practical, complete implementation of a geographically replicated database system called Detock, that enables processing strictly-serializable multi-region transactions with near-zero performance degradation at extremely high conflict and order of magnitude higher throughput relative to state-of-the art geo-replication approaches, while improving latency by up to a factor of 5.

Transaction Scheduling: From Conflicts to Runtime Conflicts

This paper studies how to improve the performance of main memory multicore OLTP systems for executing transactions with conflicts. A promising approach is to partition transaction workloads into mutually conflict-free clusters, and distribute the clusters to different cores for concurrent execution. We show that if transactions in each cluster are properly scheduled, transactions that are traditionally considered conflicting can be executed without conflicts at runtime. In light of this, we propose to schedule transactions and reduce runtime conflicts, instead of partitioning based on the conventional notion of conflicts. We formulate the transaction scheduling problem to minimize runtime conflicts, and show that the problem is NP-complete. This said, we develop an efficient scheduling algorithm to improve parallelism. Moreover, for transactions that are not packed in batches, we show that runtime conflict analysis also helps reduce conflict penalties, by proposing a proactive deferring method. Using standard and enhanced benchmarks, we show that on average our scheduling and proactive deferring methods improve the throughput of existing partitioners and concurrency control protocols by 131% and 109%, respectively, up to 294% and 152%.

Polaris: Enabling Transaction Priority in Optimistic Concurrency Control

Transaction priority is a critical feature for real-world database systems. Under high contention, certain classes of transactions should be given a higher chance to commit than others. Such a prioritization mechanism is commonly implemented in locking-based concurrency control protocols as some lock scheduling mechanisms, but it is rarely supported in the world of optimistic concurrency control. We present Polaris, an optimistic concurrency control protocol that supports multiple priority levels. To enforce priority, Polaris introduces a minimal amount of pessimism through a lightweight reservation mechanism. The protocol is fully optimistic among transactions within the same priority level and preserves the high throughput advantage of optimistic protocols. Our evaluation with YCSB workload shows that Polaris can make the p999 tail latency of high-priority transactions 13x lower than that of low-priority ones. With an abort-aware priority assignment policy, Polaris can deliver 1.9x higher throughput and 17x lower tail latency compared to Silo for high-contention workloads.

When Private Blockchain Meets Deterministic Database

Private blockchain as a replicated transactional system shares many commonalities with distributed database. However, the intimacy between private blockchain and deterministic database has never been studied. In essence, private blockchain and deterministic database both ensure replica consistency by determinism. In this paper, we present a comprehensive analysis to uncover the connections between private blockchain and deterministic database. While private blockchains have started to pursue deterministic transaction executions recently, deterministic databases have already studied deterministic concurrency control protocols for almost a decade. This motivates us to propose Harmony, a novel deterministic concurrency control protocol designed for blockchain use. We use Harmony to build a new relational blockchain, namely HarmonyBC, which features low abort rates, hotspot resiliency, and inter-block parallelism, all of which are especially important to disk-oriented blockchain. Empirical results on Smallbank, YCSB, and TPC-C show that HarmonyBC offers 2.0x to 3.5x throughput better than the state-of-the-art private blockchains.

RDMA-Enabled Concurrency Control Protocols for Transactions in the Cloud Era

Online transaction processing (OLTP) is widely used on modern cloud infrastructures to complete important businesses such as payments and stock exchanges. Remote Direct Memory Access (RDMA) is a technology that enables ultra-low inter-server memory access latency - which is critical for implementing high-performance concurrency control protocols in distributed OLTP. In this article, we develop RCC, the first unified and comprehensive RDMA-enabled distributed transaction processing framework containing six serializable concurrency control protocols—not only the classical protocols <inline-formula><tex-math notation="LaTeX">${{\sf NOWAIT}}$</tex-math></inline-formula> , <inline-formula><tex-math notation="LaTeX">${{\sf WAITDIE}}$</tex-math></inline-formula> , <inline-formula><tex-math notation="LaTeX">${{\sf OCC}}$</tex-math></inline-formula> , but also more advanced <inline-formula><tex-math notation="LaTeX">${{\sf MVCC}}$</tex-math></inline-formula> , <inline-formula><tex-math notation="LaTeX">${{\sf SUNDIAL}}$</tex-math></inline-formula> , and <inline-formula><tex-math notation="LaTeX">${{\sf CALVIN}}$</tex-math></inline-formula> — the deterministic protocol. Our goal is to unbiasedly compare protocols on OLTP workloads in a common execution environment with the concurrency control protocol being the only changeable component. From RCC, we obtained new insights on building RDMA-based protocols. We analyzed stage-wise latency breakdown to develop more efficient hybrid implementations. Moreover, RCC can enumerate all stage-wise hybrid designs under a given workload characteristic. Our results show that throughput-wise hybrid designs are better than RPC or one-sided counterparts by 32.2% and up to 67%; three hybrid designs are better than their pure counterparts by up to 17.8%. RCC can provide performance insights and be used as the common open-sourced infrastructure for fast prototyping new implementations.

Scalable Timestamp Allocation for Deadlock Prevention in Two-phase Locking based Protocols

Concurrency control ensures the correctness of databases when transactions are processed in parallel. Bamboo is a state-of-the-art concurrency control protocol extended from 2PL Wound-Wait. One problem of Bamboo is that it requires transactions to fetch timestamps from a single centralized atomic counter. To replace the concentrated access to it, each transaction should generate timestamps independently. This paper proposes three methods of decentralization to address the problem. The first is the thread-ID method (TID), which dismisses the process of fetching timestamps entirely by assigning an ID to each thread, and transactions use the thread IDs as their timestamps. In high-contention settings, the performance of TID plummets, but the second method FairTID, which is an extension of TID, sustains the performance even in such settings. The third method (RandID) allocates timestamps using random-number generators. Experiments were conducted to evaluate the performance of the proposed methods. The results indicated that there were up to 62% and 59% improvement in throughput from Bamboo with FairTID and RandID, respectively. In high-contention settings, TID underperformed Bamboo, but FairTID and RandID showed consistent improvement from Bamboo.

Intermittent-Aware Distributed Concurrency Control

Internet of Things (IoT) devices are gradually adopting battery-less, energy harvesting solutions, thereby driving the development of an intermittent computing paradigm to accumulate computation progress across multiple power cycles. While many attempts have been made to enable standalone intermittent systems, little attention has focused on IoT networks formed by intermittent devices. We observe that the computation progress improved by distributed task concurrency in an intermittent network can be significantly offset by data unavailability due to frequent system failures. This article presents an intermittent-aware distributed concurrency control protocol which leverages existing data copies inherently created in the network to improve the computation progress of concurrently executed tasks. In particular, we propose a borrowing-based data management method to increase data availability and an intermittent two-phase commit procedure incorporated with distributed backward validation to ensure data consistency in the network. The proposed protocol was integrated into a FreeRTOS-extended intermittent operating system running on Texas Instruments devices. Experimental results show that the computation progress can be significantly improved, and this improvement is more apparent under weaker power, where more devices will remain offline for longer duration.

SGPM: A coroutine framework for transaction processing

Coroutine is able to increase program concurrency and processor core utilization. However, for adapting the coroutine-to-transaction model, the existing coroutine package has the following disadvantages: (1) Additional scheduler threads incur synchronization overhead when the load between scheduler threads and worker threads is unbalanced. (2) Coroutines are swapped out periodically to prevent deadlocks, which will increase the conflict rate by adding suspended transactions. (3) Supporting only the swap-out function (yield, await, etc.) cannot flexibly control the transaction swap-in time. In this paper, we present SGPM, a coroutine framework for transaction processing. To adapt to the coroutine-to-transaction model, SGPM has the following properties: First, it eliminates scheduler threads and the periodic coroutine switch. Second, it provides a variety of coroutine scheduling strategies to make all types of concurrency control protocols run on SGPM reasonably. We implement eight well-known concurrency control on SGPM and, particularly, we use SGPM to optimize the performance of four wound-wait concurrency control among them, including 2PL, SS2PL, Calvin, and EWV. The experiment result demonstrates that after SGPM optimization 2PL and SS2PL outperform OCC and MVCC, and the throughput of Calvin and EWV is also improved by 1.2x and 1.3x respectively.

Online schema evolution is (almost) free for snapshot databases

Modern database applications often change their schemas to keep up with the changing requirements. However, support for online and transactional schema evolution remains challenging in existing database systems. Specifically, prior work often takes ad hoc approaches to schema evolution with "patches" applied to existing systems, leading to many corner cases and often incomplete functionality. Applications therefore often have to carefully schedule downtimes for schema changes, sacrificing availability. This paper presents Tesseract, a new approach to online and transactional schema evolution without the aforementioned drawbacks. We design Tesseract based on a key observation: in widely used multi-versioned database systems, schema evolution can be modeled as data modification operations that change the entire table, i.e., data-definition-as-modification (DDaM). This allows us to support schema almost "for free" by leveraging the concurrency control protocol. By simple tweaks to existing snapshot isolation protocols, on a 40-core server we show that under a variety of workloads, Tesseract is able to provide online, transactional schema evolution without service downtime, and retain high application performance when schema evolution is in progress.

C5

Asynchronously replicated primary-backup databases are commonly deployed to improve availability and offload read-only transactions. To both apply replicated writes from the primary and serve read-only transactions, the backups implement a cloned concurrency control protocol. The protocol ensures read-only transactions always return a snapshot of state that previously existed on the primary. This compels the backup to exactly copy the commit order resulting from the primary's concurrency control. Existing cloned concurrency control protocols guarantee this by limiting the backup's parallelism. As a result, the primary's concurrency control executes some workloads with more parallelism than these protocols. In this paper, we prove that this parallelism gap leads to unbounded replication lag, where writes can take arbitrarily long to replicate to the backup and which has led to catastrophic failures in production systems. We then design C5, the first cloned concurrency protocol to provide bounded replication lag. We implement two versions of C5: Our evaluation in MyRocks, a widely deployed database, demonstrates C5 provides bounded replication lag. Our evaluation in Cicada, a recent in-memory database, demonstrates C5 keeps up with even the fastest of primaries.

G-tran

Graph transaction processing poses unique challenges such as random data access due to the irregularity of graph structures, low throughput and high abort rate due to the relatively large read/write sets in graph transactions. To address these challenges, we present G-Tran, a remote direct memory access (RDMA)-enabled distributed in-memory graph database with serializable and snapshot isolation support. First, we propose a graph-native data store to achieve good data locality and fast data access for transactional updates and queries. Second, G-Tran adopts a fully decentralized architecture that leverages RDMA to process distributed transactions with the massively parallel processing (MPP) model, which can achieve high performance by utilizing all computing resources. In addition, we propose a new multi-version optimistic concurrency control (MV-OCC) protocol with two optimizations to address the issue of large read/write sets in graph transactions. Extensive experiments show that G-Tran achieves competitive performance compared with other popular graph databases on benchmark workloads.

In-page shadowing and two-version timestamp ordering for mobile DBMSs

Increasing the concurrency level in mobile database systems has not received much attention, mainly because the concurrency requirements of mobile workloads has been regarded to be low. Contrary to popular belief, mobile workloads require higher concurrency. In this work, we propose novel journaling and concurrency mechanisms for mobile DBMSs, both of which build upon one common concept - In-Page Shadowing (IPS). We design and implement a novel In-Page Shadowing recovery method for SQLite to resolve the journaling of journal anomaly, which is known to quadruple the I/O traffic in mobile devices. IPS unions the previous and the next versions of a database page in the same physical page. Using the consolidated two versions of database page, we design Two-Version Timestamp-Ordering (2VTO) protocol that enables non-blocking reads as in multi-version concurrency control, but reduces the garbage collection overhead. Designed with mobile environments in mind, IPS and 2VTO are high-performant and resource-efficient transactional solutions. Our performance study shows that IPS and 2VTO outperform state-of-the-art logging methods and an optimistic concurrency control protocol for real mobile workloads.

Opportunities for optimism in contended main-memory multicore transactions

Main-memory multicore transactional systems have achieved excellent performance using single-version optimistic concurrency control (OCC), especially on uncontended workloads. Nevertheless, systems based on other concurrency control protocols, such as hybrid OCC/ locking and variations on multiversion concurrency control (MVCC), are reported to outperform the best OCC systems, especially with increasing contention. This paper shows that implementation choices unrelated to concurrency control can explain some of these performance differences. Our evaluation shows the strengths and weaknesses of OCC, MVCC, and TicToc concurrency control under varying workloads and contention levels, and the importance of several implementation choices called basis factors. Given sensible basis factor choices, OCC performance does not collapse on high-contention TPC-C. We also present two optimization techniques, deferred updates and timestamp splitting, that can dramatically improve the high-contention performance of both OCC and MVCC. These techniques are known, but we apply them in a new context and highlight their potency: when combined, they lead to performance gains of \(4.74\times \) for MVCC and \(5.01\times \) for OCC in a TPC-C workload.

Concurrency control for real‐time and mobile transactions: Historical view, challenges, and evolution of practices

SummaryRecently, mobile computing has changed the way that spatial data and GIS are processed. Unlike wired and stand‐alone GIS, now the trend has been switched from offline to real‐time data processing using location aware services, such as GPS technology. The increased usage of location aware services in multiuser real‐time environment has made transaction management incredibly significant. If the simultaneous query operations on the same data item are not handled intelligently then this results in data inconsistency issue. Concurrency control protocol is one of the primary aspects that helps in overcoming this issue a in multiuser environment. To the best of our knowledge, the impact of technological advancements on concurrency control has not been thoroughly studied in the literature. In this article, we explored the literature on concurrency control algorithms in depth with respect to real‐time applications and the applications with moving objects. We defined a taxonomy of concurrency control solutions and assessed the maturity of these solutions in the light of characteristics of real‐time and mobile environment. We compared the most recent developments made in the literature and presented meaningful insights. Challenges are also identified and discussed, which can assist in doing research in this domain in future.

Hyperspace

Microsoft recently introduced Azure Synapse Analytics, which offers an integrated experience across data ingestion, storage, and querying in Apache Spark and T-SQL over data in the lake, including files and warehouse tables. In this paper, we present our experiences with designing and implementing Hyperspace, the indexing subsystem underlying Synapse. Hyperspace enables users to build multiple types of secondary indexes on their data, maintain them through a multi-user concurrency model, and leverage them automatically---without any change to their application code---for query/workload acceleration. Many requirements of Hyperspace are based on feedback from several enterprise customers. We present the details of Hyperspace's underlying design, the user-facing APIs, its concurrency control protocol for index access, its index-aware query processing techniques, and its maintenance mechanisms for handling index updates. Evaluations over standard industry benchmarks and real customer workloads show that Hyperspace can accelerate query execution by up to 10x and in certain real-world workloads, even up to two orders of magnitude.

A High Performance Concurrency Protocol for Smart Contracts of Permissioned Blockchain

Although the emergence of the programmable smart contract makes blockchain systems easily embrace a wide range of industrial services, how to execute smart contracts efficiently becomes a big challenge nowadays. Due to the existence of Byzantine nodes, existing mature concurrency control protocols in database cannot be employed directly, since the mechanism of executing smart contracts varies a lot. Furthermore, even though smart contract execution follows a two-phase style, i.e., the primary node executes a batch of smart contracts in the first phase and the validators replay them in the second phase, existing parallel solutions merely focus on the optimization for the first phase, rather than the second phase. In this paper, we propose a novel two-phase concurrency control protocol to optimize both phases for the first time. First, the primary executes transactions in parallel and generates a transaction dependency graph with high parallelism for validators. Then, a graph partition algorithm is devised to divide the original graph into several sub-graphs to preserve parallelism and reduce communication cost remarkably. Finally, we propose a deterministic replay protocol to re-execute the primary’s parallel schedule concurrently. Moreover, this two-phase protocol is further optimized by integrating with PBFT. Theoretical analysis and extensive experimental results illustrate that the proposed scheme outperforms state-of-art solutions significantly.