Serializability Of Transactions Research Articles

SconeKV: A Scalable, Strongly Consistent Key-Value Store

For decades, relational databases provided a strong foundation for constructing applications due to their ACID properties. However, distributed applications reached a scale, both in terms of data volume and number of concurrent clients, that traditional databases cannot accommodate. NoSQL databases addressed this problem by trading consistency for scalability, namely through horizontal scalability schemes supported by optimistic replication protocols, which only guarantee eventual consistency. In this paper, we explore a novel design between the two extremes, which is able to scale to large deployments while still offering strong consistency guarantees in the form of serializable transactions. Our key insight is to leverage recent advances in membership services that provide strongly consistent views at scale. Those assurances from the membership layer simplify building efficient and consistent storage protocols. Our evaluation of the resulting system, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">SconeKV</small> , in a realistic scenario shows that it scales and performs better than CockroachDB while being competitive with Cassandra.

A Bilayer Scalable Nakamoto Consensus Protocol for Blockchain Systems

How to simultaneously achieve high security level, good scalability, and full decentralization is still a big issue in the design of a blockchain system. In this article, we propose Bicomp, a bilayer, scalable Nakamoto consensus protocol that is based purely on decentralized Nakamoto consensus. Bicomp works to generate two types of blocks - microblocks for concurrent in-network transaction serialization and macroblocks for chain formation. A leader is elected at the beginning of each round through Proof of Work-based computation; it puts microblocks mined by different nodes into one macroblock at the end of its tenure. Such bilayer design limits a leader's power and encourages as many nodes as possible to concurrently participate in the transaction serialization process, promoting the sharding nature of the system. Furthermore, several mechanisms are elaborately designed to reduce the transaction overlapping and further limit a leader's power, including a novel transaction diversity-based metric for deciding the branch legality. Security issues and potential attacks to Bicomp are extensively discussed. Experimental results based on 500 nodes all over the world show that Bicomp achieves high throughput performance.

Incentive Analysis of Bitcoin-NG, Revisited

Bitcoin-NG is among the first scalable blockchain protocols by decoupling blockchain operation into two planes: leader election and transaction serialization. Its decoupling idea has inspired a new generation of blockchain protocols. However, the existing incentive analysis of Bitcoin-NG has several limitations. First, the impact of network capacity is ignored. Second, an integrated incentive analysis that jointly considers both key blocks and microblocks is still missing. In this paper, we aim to address these two limitations. First, we propose a new incentive analysis that takes the network capacity into account, showing that Bitcoin-NG can still maintain incentive compatibility against the microblock mining attack even under limited network capacity. Second, we leverage a Markov decision process to jointly analyze the incentive of both key blocks and microblocks, showing that the selfish mining revenue of Bitcoin-NG is a little higher than that in Bitcoin only when the selfish miner controls more than 35% of the mining power. We hope that our in-depth incentive analysis for Bitcoin-NG can shed some light on the mechanism design and incentive analysis of nextgeneration blockchain protocols.

Incentive analysis of Bitcoin-NG, revisited

Bitcoin-NG is among the first blockchain protocols to approach the near-optimal throughput by decoupling blockchain operation into two planes: leader election and transaction serialization. Its decoupling idea has inspired a new generation of high-performance blockchain protocols. However, the existing incentive analysis of Bitcoin-NG has several limitations. First, the impact of network capacity is ignored. Second, an integrated incentive analysis that jointly considers both key blocks and microblocks is still missing.In this paper, we aim to address these two limitations. First, we propose a new incentive analysis that takes the network capacity into account, showing that Bitcoin-NG can still maintain incentive compatibility against the microblock mining attack even under limited network capacity. Second, we leverage a Markov decision process (MDP) to jointly analyze the incentive of both key blocks and microblocks, showing that the selfish mining revenue of Bitcoin-NG is a little higher than that in Bitcoin only when the selfish miner controls more than 35% of the mining power. We hope that our in-depth incentive analysis for Bitcoin-NG can shed some light on the mechanism design and incentive analysis of next-generation blockchain protocols.

Energy-Efficient Transaction Serialization for IoT Devices

This article presents two designs, the Transaction Serial Format (TSF) and the Transaction Array Model (TAM). Together, they provide full, efficient, transaction serialization facilities for devices with limited onboard energy, such as those in an Internet of Things (IoT) network. TSF provides a compact, non-parsed, format that requires minimal processing for transaction deserialization. TAM provides an internal data structure that needs minimal dynamic storage and directly uses the elements of TSF. The simple lexical units of TSF do not require parsing. The lexical units contain enough information to allocate the internal TAM data structure efficiently. TSF generality is equivalent to XML and JSON. TSF represents any XML document or JSON object without loss of information, including whitespace. The XML equivalence provides a foundation for the performance comparisons. A performance comparison of a C reference implementation of TSF and TAM to the popular Expat XML library, also written in C, shows that TSF reduces deserialization processor time by more than 80%.

STAR

In this paper, we present STAR, a new distributed in-memory database with asymmetric replication. By employing a single-node non-partitioned architecture for some replicas and a partitioned architecture for other replicas, STAR is able to efficiently run both highly partitionable workloads and workloads that involve cross-partition transactions. The key idea is a new phase-switching algorithm where the execution of single-partition and cross-partition transactions is separated. In the partitioned phase, single-partition transactions are run on multiple machines in parallel to exploit more concurrency. In the single-master phase, mastership for the entire database is switched to a single designated master node, which can execute these transactions without the use of expensive coordination protocols like two-phase commit. Because the master node has a full copy of the database, this phase-switching can be done at negligible cost. Our experiments on two popular benchmarks (YCSB and TPC-C) show that high availability via replication can coexist with fast serializable transaction execution in distributed in-memory databases, with STAR outperforming systems that employ conventional concurrency control and replication algorithms by up to one order of magnitude.

MixT: a language for mixing consistency in geodistributed transactions

Programming concurrent, distributed systems is hard—especially when these systems mutate shared, persistent state replicated at geographic scale. To enable high availability and scalability, a new class of weakly consistent data stores has become popular. However, some data needs strong consistency. To manipulate both weakly and strongly consistent data in a single transaction, we introduce a new abstraction: mixed-consistency transactions, embodied in a new embedded language, MixT. Programmers explicitly associate consistency models with remote storage sites; each atomic, isolated transaction can access a mixture of data with different consistency models. Compile-time information-flow checking, applied to consistency models, ensures that these models are mixed safely and enables the compiler to automatically partition transactions. New run-time mechanisms ensure that consistency models can also be mixed safely, even when the data used by a transaction resides on separate, mutually unaware stores. Performance measurements show that despite their stronger guarantees, mixed-consistency transactions retain much of the speed of weak consistency, significantly outperforming traditional serializable transactions.

Scaling distributed transaction processing and recovery based on dependency logging

Dependency graph-based concurrency control (DGCC) protocol has been shown to achieve good performance on multi-core in-memory system. DGCCseparates contention resolution from the transaction execution and employs dependency graphs to derive serializable transaction schedules. However, distributed transactions complicate the dependency resolution, and therefore, an effective transaction partitioning strategy is essential to reduce expensive multi-node distributed transactions. During failure recovery, log must be examined from the last checkpoint onward and the affected transactions are re-executed based on the way they are partitioned and executed. Existing approaches treat both transaction management and recovery as two separate problems, even though recovery is dependent on the sequence in which transactions are executed. In this paper, we propose to treat the transaction management and recovery problems as one. We first propose an efficient distributed dependency graph-based concurrency control (DistDGCC) protocol for handling transactions spanning multiple nodes and propose a new novel and efficient logging protocol called dependency logging that also makes use of dependency graphs for efficient logging and recovery. DistDGCCoptimizes the average cost for each distributed transaction by processing transactions in batches. Moreover, it also reduces the effects of thread blocking caused by distributed transactions and consequently improves the runtime performance. Further, dependency logging exploits the same data structure that is used by DistDGCCto reduce the logging overhead, as well as the logical dependency information to improve the recovery parallelism. Extensive experiments are conducted to evaluate the performance of our proposed technique against state-of-the-art techniques. Experimental results show that DistDGCCis efficient and scalable, and dependency logging supports fast recovery with marginal runtime overhead. Hence, the overall system performance is significantly improved as a result.

Alone together: compositional reasoning and inference for weak isolation

Serializability is a well-understood correctness criterion that simplifies reasoning about the behavior of concurrent transactions by ensuring they are isolated from each other while they execute. However, enforcing serializable isolation comes at a steep cost in performance because it necessarily restricts opportunities to exploit concurrency even when such opportunities would not violate application-specific invariants. As a result, database systems in practice support, and often encourage, developers to implement transactions using weaker alternatives. These alternatives break the strong isolation guarantees offered by serializable transactions to permit greater concurrency. Unfortunately, the semantics of weak isolation is poorly understood, and usually explained only informally in terms of low-level implementation artifacts. Consequently, verifying high-level correctness properties in such environments remains a challenging problem. To address this issue, we present a novel program logic that enables compositional reasoning about the behavior of concurrently executing weakly-isolated transactions. Recognizing that the proof burden necessary to use this logic may dissuade application developers, we also describe an inference procedure based on this foundation that ascertains the weakest isolation level that still guarantees the safety of high-level consistency assertions associated with such transactions. The key to effective inference is the observation that weakly-isolated transactions can be viewed as functional (monadic) computations over an abstract database state, allowing us to treat their operations as state transformers over the database. This interpretation enables automated verification using off-the-shelf SMT solvers. Our development is parametric over a transaction’s specific isolation semantics, allowing it to be applicable over a range of concurrency control mechanisms. Case studies and experiments on real-world applications (written in an embedded DSL in OCaml) demonstrate the utility of our approach, and provide strong evidence that automated verification of weakly-isolated transactions can be placed on the same formal footing as their strongly-isolated serializable counterparts.

BCC

The Optimistic Concurrency Control (OCC) method has been commonly used for in-memory databases to ensure transaction serializability --- a transaction will be aborted if its read set has been changed during execution. This simple criterion to abort transactions causes a large proportion of false positives, leading to excessive transaction aborts. Transactions aborted false-positively (i.e. false aborts) waste system resources and can significantly degrade system throughput (as much as 3.68x based on our experiments) when data contention is intensive. Modern in-memory databases run on systems with increasingly parallel hardware and handle workloads with growing concurrency. They must efficiently deal with data contention in the presence of greater concurrency by minimizing false aborts. This paper presents a new concurrency control method named Balanced Concurrency Control (BCC) which aborts transactions more carefully than OCC does. BCC detects data dependency patterns which can more reliably indicate unserializable transactions than the criterion used in OCC. The paper studies the design options and implementation techniques that can effectively detect data contention by identifying dependency patterns with low overhead. To test the performance of BCC, we have implemented it in Silo and compared its performance against that of the vanilla Silo system with OCC and two-phase locking (2PL). Our extensive experiments with TPC-W-like, TPC-C-like and YCSB workloads demonstrate that when data contention is intensive, BCC can increase transaction throughput by more than 3x versus OCC and more than 2x versus 2PL; meanwhile, BCC has comparable performance with OCC for workloads with low data contention.

Determinism at Standard-Library Level in TM-Based Applications

Deterministic execution of a multi-threaded application guarantees that threads access shared memory in the same order and the application gives the same output whenever it runs with the same input parameters. Determinism provides repeatability, which helps programmers in testing and debugging. Additionally, transactional memory (TM) simplifies development of multi-threaded applications so that programmers can use transactions (instead of locks) to synchronize accesses to shared memory. However, transactions that call standard library functions have to be serialized, i.e. to be executed as the only running transactions in the system. The serialization enforces an order of threads execution usually different from the one enforced by a system for deterministic multithreading, which causes deadlocks in the application execution. In this paper, we present DeTrans-lib, the first standard C library that provides deterministic execution of TM-based applications at application and standard-library level. DeTrans-lib avoids deadlocks by performing transaction serialization in deterministic order. We evaluate DeTrans-lib with the benchmarks that invoke a standard C library and perform I/O operations.

Rethinking serializable multiversion concurrency control

Multi-versioned database systems have the potential to significantly increase the amount of concurrency in transaction processing because they can avoid read-write conflicts. Unfortunately, the increase in concurrency usually comes at the cost of transaction serializability. If a database user requests full serializability, modern multi-versioned systems significantly constrain read-write concurrency among conflicting transactions and employ expensive synchronization patterns in their design. In main-memory multi-core settings, these additional constraints are so burdensome that multi-versioned systems are often significantly outperformed by single-version systems. We propose B ohm , a new concurrency control protocol for main-memory multi-versioned database systems. B ohm guarantees serializable execution while ensuring that reads never block writes. In addition, B ohm does not require reads to perform any bookkeeping whatsoever, thereby avoiding the overhead of tracking reads via contended writes to shared memory. This leads to excellent scalability and performance in multi-core settings. B ohm has all the above characteristics without performing validation based concurrency control. Instead, it is pessimistic, and is therefore not prone to excessive aborts in the presence of contention. An experimental evaluation shows that B ohm performs well in both high contention and low contention settings, and is able to dramatically outperform state-of-the-art multi-versioned systems despite maintaining the full set of serializability guarantees.

Bumper: Sheltering distributed transactions from conflicts

Large scale cloud applications are difficult to program due to the need to access data in a consistent manner. To lift this burden from programmers, Deferred Update Replication (DUR) protocols provide serializable transactions with both high availability and performance in read-dominated workloads. However, the inherently optimistic nature of DUR protocols makes them prone to thrashing in conflict-intensive scenarios: existing DUR schemes, in fact, avoid any synchronization during transaction execution; thus, these schemes end up aborting any update transaction whose reads are no longer up to date by the time it attempts to commit.To tackle this problem, we introduce Bumper, a set of innovative techniques to reduce aborts of transactions in high-contention scenarios. At its core, Bumper relies on two key ideas. First, we spare update transactions from spurious aborts (i.e., unnecessary aborts of serializable transactions), by attempting to serialize the transactions in the past. For this, we use a novel distributed concurrency control scheme that we call Distributed Time-Warping (DTW). And second, we avoid aborts due to contention hot spots (that cannot be tackled by DTW) via a programming abstraction that we call Delayed Actions. These, allow for efficiently serializing, in an abort-free fashion, the execution of conflict-prone data manipulations.By means of an extensive evaluation, on a distributed system of 160 nodes, we show that Bumper can boost performance up to 3× in conflict-intensive workloads, while imposing negligible (about 2.5%) overheads in uncontended scenarios.

Coordination avoidance in database systems

Minimizing coordination, or blocking communication between concurrently executing operations, is key to maximizing scalability, availability, and high performance in database systems. However, uninhibited coordination-free execution can compromise application correctness, or consistency. When is coordination necessary for correctness? The classic use of serializable transactions is sufficient to maintain correctness but is not necessary for all applications, sacrificing potential scalability. In this paper, we develop a formal framework, invariant confluence, that determines whether an application requires coordination for correct execution. By operating on application-level invariants over database states (e.g., integrity constraints), invariant confluence analysis provides a necessary and sufficient condition for safe, coordination-free execution. When programmers specify their application invariants, this analysis allows databases to coordinate only when anomalies that might violate invariants are possible. We analyze the invariant confluence of common invariants and operations from real-world database systems (i.e., integrity constraints) and applications and show that many are invariant confluent and therefore achievable without coordination. We apply these results to a proof-of-concept coordination-avoiding database prototype and demonstrate sizable performance gains compared to serializable execution, notably a 25-fold improvement over prior TPC-C New-Order performance on a 200 server cluster.

Selective dynamic serialization for reducing energy consumption in hardware transactional memory systems

In the search for new paradigms to simplify multithreaded programming, Transactional Memory (TM) is currently being advocated as a promising alternative to deadlock-prone lock-based synchronization. In this way, future many-core CMP architectures may need to provide hardware support for TM. On the other hand, power dissipation constitutes a first class consideration in multicore processor designs. In this work, we propose Selective Dynamic Serialization (SDS) as a new technique to improve energy consumption without degrading performance in applications with conflicting transactions by avoiding wasted work due to aborted transactions. Our proposal, which is implemented on top of a hardware transactional memory (HTM) system with an eager conflict management policy, detects and serializes conflicting transactions dynamically (at run-time). In its simplest form, in case of conflict, one transaction is allowed to continue whilst the rest are completely stalled. Once the executing transaction has finished, it wakes up several of the stalling transactions. More elaborated implementations of SDS try to delay this behavior until serialization of transactions is profitable, achieving the best trade-off between performance, energy savings and network traffic. SDS implementations differ from each other in the condition that triggers the serialization mode. We have evaluated several SDS schemes using GEMS, a full-system simulator implementing the LogTM-SE Eager---Eager HTM system, and several benchmarks from the STAMP suite. Results for a 16-core CMP show that SDS obtains reductions of 6 % on average in energy consumption (more than 20 % in high contention scenarios) in a wide range of benchmarks without affecting, on average, execution time. At the same time, network traffic level is also reduced by 22 %.

On the impact of serializing contention management on STM performance

Transactional memory (TM) is an emerging concurrent programming abstraction. Numerous software-based transactional memory (STM) implementations have been developed in recent years. STM implementations must guarantee transaction atomicity and isolation. In order to ensure progress, an STM implementation must resolve transaction collisions by consulting a contention manager (CM).Recent work established that serializing contention management–a technique in which the execution of colliding transactions is serialized for eliminating repeat-collisions–can dramatically improve STM performance in high-contention workloads. In low-contention and highly-parallel workloads, however, excessive serialization of memory transactions may limit concurrency too much and hurt performance. It is therefore important to better understand how the impact of serialization on STM performance varies as a function of workload characteristics.We investigate how serializing CM influences the performance of STM systems. Specifically, we study serialization’s influence on STM throughput (number of committed transactions per time unit) and efficiency (ratio between the extent of “useful” work done by the STM and work “wasted” by aborts) as the workload’s level of contention changes. Towards this goal, we implement CBench—a synthetic benchmark that generates workloads in which transactions have (parameter) pre-determined length and probability of being aborted in the lack of contention reduction mechanisms. CBench facilitates evaluating the efficiency of contention management algorithms across the full spectrum of contention levels.The characteristics of TM workloads generated by real applications may vary over time. To achieve good performance, CM algorithms need to monitor these characteristics and change their behavior accordingly. We implement adaptive algorithms that control the activation of serializing CM according to measured contention level, based on a novel low-overhead serialization mechanism. We then evaluate our new algorithms on CBench-generated workloads and on additional well-known STM benchmark applications.Our results shed light on the manner in which serializing CM should be used by STM systems. We show that adaptive contention managers are susceptible to a phenomenon of mode oscillations–in which serialization is repeatedly turned on and off–which hurts performance. We implement a simple stabilizing mechanism that solves this problem. We also compare the performance of local and global adaptive CM algorithms and demonstrate that local adaptive algorithms are superior for applications with asymmetric workloads.

Cooperative reasoning for preemptive execution

We propose a cooperative methodology for multithreaded software, where threads use traditional synchronization idioms such as locks, but additionally document each point of potential thread interference with a "yield" annotation. Under this methodology, code between two successive yield annotations forms a serializable transaction that is amenable to sequential reasoning. This methodology reduces the burden of reasoning about thread interleavings by indicating only those interference points that matter. We present experimental results showing that very few yield annotations are required, typically one or two per thousand lines of code. We also present dynamic analysis algorithms for detecting cooperability violations, where thread interference is not documented by a yield, and for yield annotation inference for legacy software.

Global Timestamp Serialization in Multi-Level Multi-Version DBMS

The concurrency control mechanism in the multi-level DBMS is required to promise the serializability of transactions and the multi-level security properties, avoid possible covert channels and the starving problem of high-level transactions. The multi-level multi-version timestamp ordering mechanisms satisfy these requirements but may cause transactions read old version data, and the scheduler is required to be a trusted process. This paper presents a multi-level multi-version global timestamp ordering (MLS-MVGTO) mechanism and the basic global timestamps generation steps based on the transaction’s snapshot. This paper also presents two improvements according to the pre-knowledge of the read-only transactions. In addition it can be implemented as a set of untrusted schedulers. Given the pre-knowledge about transactions’ operations, transactions are able to read more recent version.

On consistent reading of entire databases

Many applications need to read an entire database in a consistent way. This global-reading of an entire database formulated as a global-read transaction (GRT) is not a trivial issue since it will cause a high degree of interference to other concurrent transactions. Conventional concurrency control protocols are obviously inadequate in handling the long-lived GRT. Previous studies proposed additional tests, namely, the color test and the shade test, to handle conflicts between the GRT and update transactions. However, we discovered that both algorithms can bring about nonserializable schedules of transactions. We propose an enhanced algorithm directly built on the two algorithms to guarantee the serializability of transactions.

A new primary lazy update propagation protocol for replicated databases

This paper proposes a new primary lazy update protocol, PTCS (Primary Transaction Commit Schedule). In the PTCS protocol, a serializable primary transaction schedule is generated firstly and then the secondary transactions are committed according to the serializable primary transaction schedule. PTCS protocol can guarantee serializability if the data copy graph contains no directed circles. It can also be extended to eliminate all requirements on the data copy graph. Compared to earlier works, PTCS protocol not only imposes a much weaker requirement on the data placement, but also avoids the deadlock caused by transaction waits and extra message overhead. The performance experiments show that the degradation of the performance caused by the replica management of the PTCS protocol is tolerable.