ACID Transactions Research Articles

Cloud Actor-Oriented Database Transactions in Orleans

Microsoft Orleans is a popular open source distributed programming framework and platform which invented the virtual actor model, and has since evolved into an actor-oriented database system with the addition of database abstractions such as ACID transactions. Properties of Orleans' virtual actor model imply that any ACID transaction mechanism for operations spanning multiple actors must support distributed transactions on top of pluggable cloud storage drivers. Unfortunately, distributed transactions usually perform poorly in this environment, partly because of the high performance and contention overhead of performing two-phase commit (2PC) on slow cloud storage systems. In this paper we describe the design and implementation of ACID transactions in Orleans. The system uses two primary techniques to mask the high latency of cloud storage and enable high transaction throughput. First, Orleans pioneered the use of a distributed form of early lock release by releasing all of a transaction's locks during phase one of 2PC, and by tracking commit dependencies to implement cascading abort. This avoids blocking transactions while running 2PC and enables a distributed form of group commit. Second, Orleans leverages reconnaissance queries to prefetch the state of all actors involved in a transaction from cloud storage prior to running the transaction and acquiring any locks, thus ensuring no locks are held while blocking on high latency cloud storage in most cases.

PALF: Replicated Write-Ahead Logging for Distributed Databases

Distributed databases have been widely researched and developed in recent years due to their scalability, availability, and consistency guarantees. The write-ahead logging (WAL) system is one of the most vital components in a database. It is still a non-trivial problem to design a replicated logging system as the foundation of a distributed database with the power of ACID transactions. This paper proposes PALF, a Paxos-backed Append-only Log File System, to address these challenges. The basic idea behind PALF is to co-design the logging system with the entire database for supporting database-specific functions and to abstract the functions as PALF primitives to power other distributed systems. Many database functions, including transaction processing, database restore, and physical standby databases, have been built based on PALF primitives. Evaluation shows that PALF greatly outperforms well-known implementations of consensus protocols and is fully competent for distributed database workloads. PALF has been deployed as a component of the OceanBase 4.0 database and has been made open-source along with it.

An approach for supporting transparent acid transactions over heterogeneous data stores in microservice architectures

Microservice architectures (MSA) are becoming a preferred architectural style for data-driven applications. A transaction within MSA can include remote calls to multiple services, turning it into a distributed transaction. Participating services may have their own data stores running local transactions with varying levels of transactional support and consistency guarantees. Coordinating distributed transactions in such an environment is a key challenge for MSA. The existing approaches are either highly consistent at the expense of scalability or scalable at the expense of consistency. Furthermore, implementing any of them requires architectural and code adaptation. In this article, we present the Service Proxy Transaction Management (SPTM) approach, which offers scalable reads and ACID transactions in MSA. The novelty of this approach is that it is based on intercepting inbound messages to services, rather than having services directly communicate with a transaction manager. As a result, transaction management is completely transparent to services and has little-to-no impact on code or architecture. We provide experimental results showing that SPTM can outperform lock-based approaches by up to a factor of 2, while still providing high consistency without the scaling bottleneck associated with locking.

FoundationDB: A Distributed Key-Value Store

FoundationDB is an open-source transactional key-value store created more than 10 years ago. It is one of the first systems to combine the flexibility and scalability of NoSQL architectures with the power of ACID transactions. FoundationDB adopts an unbundled architecture that decouples an in-memory transaction management system, a distributed storage system, and a built-in distributed configuration system. Each sub-system can be independently provisioned and configured to achieve scalability, high availability, and fault tolerance. FoundationDB includes a deterministic simulation framework, used to test every new feature under a myriad of possible faults. This rigorous testing makes FoundationDB extremely stable and allows developers to introduce and release new features in a rapid cadence. FoundationDB offers a minimal and carefully chosen feature set, which has enabled a range of disparate systems to be built as layers on top. FoundationDB is the underpinning of cloud infrastructure at Apple, Snowflake, and other companies, due to its consistency, robustness, and availability for storing user data, system metadata and configuration, and other critical information.

Enhancing ETL Performance Using Delta Lake in Data Analytics Solutions

In the rapidly evolving field of data analytics, the performance of Extract, Transform, Load (ETL) processes is crucial for effective data management and insight generation. This study explores the integration of Delta Lake within ETL frameworks to enhance performance and reliability. Delta Lake, an open-source storage layer, facilitates ACID transactions, scalable metadata handling, and unifies batch and streaming data processing, addressing common challenges associated with traditional ETL processes. By leveraging Delta Lake’s capabilities, organizations can optimize data ingestion and transformation workflows, resulting in reduced latency and improved data quality. This research employs a comparative analysis of traditional ETL methods and those enhanced with Delta Lake, measuring key performance indicators such as processing speed, resource utilization, and error rates. Case studies illustrate the practical applications of Delta Lake in diverse industries, demonstrating its potential to streamline ETL operations while ensuring data consistency and reliability. Additionally, the study discusses the implications of adopting Delta Lake for businesses seeking to harness large-scale data for analytics and decision-making. Ultimately, this investigation highlights the transformative impact of Delta Lake on ETL performance, advocating for its adoption as a standard practice in data analytics solutions. By enhancing ETL processes, organizations can derive actionable insights more efficiently, driving innovation and competitive advantage in the data-driven landscape.

Identification of monolith functionality refactorings for microservices migration

AbstractIn the process of migrating a monolith to a microservice architecture, functionalities which execute on top of a single database as ACID transactions, have their execution split between several databases, one for each microservice. Therefore, the complexity of functionality migration increases, because of the lack of isolation of the functionality execution, which has to be handled by following an eventual consistency model. On the other hand, the object‐oriented approach, commonly followed in the development of monolith systems, promotes fine‐grained interactions in the functionalities implementation, which further increases the migration effort due to the large number of remote invocations between microservices, because it increases the number of microservices transactions that have to eventually be consistent. The main contribution of this paper is a heuristic method to help the software architect on the identification of functionality refactorings, to reduce the effort required to migrate the functionality to microservices when applying the SAGA pattern in the functionality microservices implementation. The heuristic accuracy and efficiency of the method is evaluated using a dataset extracted from 78 codebases and by comparing the results with expert refactorizations. Additionally, a performance analysis of the heuristic implementation is done.

Genetic and biochemical interactions of yeast DNA helicases.

DNA helicases function in many types of nucleic acid transactions, and as such, they are vital for genome integrity. Although they are often considered individually, work from many groups demonstrates that these enzymes often genetically and biochemically interact in vivo. Here, we highlight methods to interrogate such interactions among the PIF1 (Pif1 and Rrm3) and RecQ (Hrq1 and Sgs1) family helicases in Saccharomyces cerevisiae. The interactions among these enzymes were investigated in vivo using deletion and inactivation alleles with a gross-chromosomal rearrangement (GCR) assay. Further, wild-type and inactive recombinant proteins were used to determine the effects of the helicases on telomerase activity in vitro. We found that synergistic increases in GCR rates often occur in double vs. single mutants, suggesting that the helicases function in distinct genome integrity pathways. Further, the recombinant helicases can function together in vitro to modulate telomerase activity. Overall, the data suggest that the interactions among the members of these DNA helicase families are multipartite and argue for a comprehensive systems biology approach to fully elucidate the physiological interplay between these enzymes.

FoundationDB

FoundationDB is an open source transactional key value store created more than ten years ago. It is one of the first systems to combine the flexibility and scalability of NoSQL architectures with the power of ACID transactions. FoundationDB adopts an unbundled architecture that decouples an in-memory transaction management system, a distributed storage system, and a built-in distributed configuration system. Each sub-system can be independently provisioned and configured to achieve scalability, high-availability and fault tolerance. FoundationDB includes a deterministic simulation framework, used to test every new feature under a myriad of possible faults. FoundationDB offers a minimal and carefully chosen feature set, which has enabled a range of disparate systems to be built as layers on top. FoundationDB is the underpinning of cloud infrastructure at Apple, Snowflake and other companies.

Elastic scalable transaction processing in LeanXcale

Scaling ACID transactions in a cloud database is hard, and providing elastic scalability even harder. In this paper, we present our solution for elastic scalable transaction processing in LeanXcale, an industrial-strength NewSQL database system. Unlike previous solutions, it does not require any hardware assistance. Yet, it does scales linearly to 100s of servers. LeanXcale supports non-intrusive elasticity and can move data partitions without hurting the quality of service of transaction management. We show the correctness of LeanXcale transaction management. Finally, we provide a thorough performance evaluation of our solution on Amazon Web Services (AWS) shared cloud instances. The results show linear scalability, e.g., 5 million TPC-C NewOrder TPM with 200 nodes, which is greater than the TPC-C throughput obtained by the 9th highest result in all history using dedicated hardware used exclusively (not shared like in our evaluation) for the benchmark. Furthermore, the efficiency in terms of TPM per core is double that of the two top TPC-C results (also the only results in a cloud).

RESEARCH OF ACID TRANSACTION IMPLEMENTATION METHODS FOR DISTRIBUTED DATABASES USING REPLICATION TECHNOLOGY

Today, databases are an integral part of most modern applications designed to store large amounts of data and to request from many users. To solve business problems in such conditions, databases are scaled, often horizontally on several physical servers using replication technology. At the same time, many business operations require the implementation of transactional compliance with ACID properties. For relational databases that traditionally support ACID transactions, horizontal scaling is not always effective due to the limitations of the relational model itself. Therefore, there is an applied problem of efficient implementation of ACID transactions for horizontally distributed databases. The subject matter of the study is the methods of implementing ACID transactions in distributed databases, created by replication technology. The goal of the work is to increase the efficiency of ACID transaction implementation for horizontally distributed databases. The work is devoted to solving the following tasks: analysis and selection of the most relevant methods of implementation of distributed ACID transactions; planning and experimental research of methods for implementing ACID transactions by using of NoSQL DBMS MongoDB and NewSQL DBMS VoltDB as an example; measurements of metrics of productivity of use of these methods and formation of the recommendation concerning their effective use. The following methods are used: system analysis; relational databases design; methods for evaluating database performance. The following results were obtained: experimental measurements of the execution time of typical distributed transactions for the subject area of e-commerce, as well as measurements of the number of resources required for their execution; revealed trends in the performance of such transactions, formed recommendations for the methods studied. The obtained results allowed to make functions of dependence of the considered metrics on loading parameters. Conclusions: the strengths and weaknesses of the implementation of distributed ACID transactions using MongoDB and VoltDB were identified. Practical recommendations for the effective use of these systems for different types of applications, taking into account the resources consumed and the types of requests.

CoV-er all the bases: Structural perspectives of SARS-CoV-2 RNA synthesis.

The ongoing Covid-19 pandemic has spurred research in the biology of the nidovirus severe acute respiratory syndrome coronavirus-2 (SARS-CoV-2). Much focus has been on the viral RNA synthesis machinery due to its fundamental role in viral propagation. The central and essential enzyme of the RNA synthesis process, the RNA-dependent RNA polymerase (RdRp), functions in conjunction with a coterie of viral-encoded enzymes that mediate crucial nucleic acid transactions. Some of these enzymes share common features with other RNA viruses, while others play roles unique to nidoviruses or CoVs. The RdRps are proven targets for viral pathogens, and many of the other nucleic acid processing enzymes are promising targets. The purpose of this review is to summarize recent advances in our understanding of the mechanisms of RNA synthesis in CoVs. By reflecting on these studies, we hope to emphasize the remaining gaps in our knowledge. The recent onslaught of structural information related to SARS-CoV-2 RNA synthesis, in combination with previous structural, genetic and biochemical studies, have vastly improved our understanding of how CoVs replicate and process their genomic RNA. Structural biology not only provides a blueprint for understanding the function of the enzymes and cofactors in molecular detail, but also provides a basis for drug design and optimization. The concerted efforts of researchers around the world, in combination with the renewed urgency toward understanding this deadly family of viruses, may eventually yield new and improved antivirals that provide relief to the current global devastation.

Gossip-based visibility control for high-performance geo-distributed transactions

Providing ACID transactions under conflicts across globally distributed data is the Everest of transaction processing protocols. Transaction processing in this scenario is particularly costly due to the high latency of cross-continent network links, which inflates concurrency control and data replication overheads. To mitigate the problem, we introduce Ocean Vista—a novel distributed protocol that guarantees strict serializability. We observe that concurrency control and replication address different aspects of resolving the visibility of transactions, and we address both concerns using a multi-version protocol that tracks visibility using version watermarks and arrives at correct visibility decisions using efficient gossip. Gossiping the watermarks enables asynchronous transaction processing and acknowledging transaction visibility in batches in the concurrency control and replication protocols, which improves efficiency under high cross-data center network delays. In particular, Ocean Vista can access conflicting transactions in parallel and supports efficient write-quorum/read-one access using one round trip in the common case. We demonstrate experimentally in a multi-data center cloud environment that our design outperforms a leading distributed transaction processing engine (TAPIR) more than tenfold in terms of peak throughput, albeit at the cost of additional latency for gossip and a more restricted transaction model. The latency penalty is generally bounded by one wide area network (WAN) round trip time (RTT), and in the best case (i.e., under light load) our system nearly breaks even with TAPIR by committing transactions in around one WAN RTT.

Delta lake

Cloud object stores such as Amazon S3 are some of the largest and most cost-effective storage systems on the planet, making them an attractive target to store large data warehouses and data lakes. Unfortunately, their implementation as key-value stores makes it difficult to achieve ACID transactions and high performance: metadata operations such as listing objects are expensive, and consistency guarantees are limited. In this paper, we present Delta Lake, an open source ACID table storage layer over cloud object stores initially developed at Databricks. Delta Lake uses a transaction log that is compacted into Apache Parquet format to provide ACID properties, time travel, and significantly faster metadata operations for large tabular datasets (e.g., the ability to quickly search billions of table partitions for those relevant to a query). It also leverages this design to provide high-level features such as automatic data layout optimization, upserts, caching, and audit logs. Delta Lake tables can be accessed from Apache Spark, Hive, Presto, Redshift and other systems. Delta Lake is deployed at thousands of Databricks customers that process exabytes of data per day, with the largest instances managing exabyte-scale datasets and billions of objects.

Functional Diversification of Replication Protein A Paralogs and Telomere Length Maintenance in Arabidopsis.

Replication protein A (RPA) is essential for many facets of DNA metabolism. The RPA gene family expanded in Arabidopsis thaliana with five phylogenetically distinct RPA1 subunits (RPA1A-E), two RPA2 (RPA2A and B), and two RPA3 (RPA3A and B). RPA1 paralogs exhibit partial redundancy and functional specialization in DNA replication (RPA1B and RPA1D), repair (RPA1C and RPA1E), and meiotic recombination (RPA1A and RPA1C). Here, we show that RPA subunits also differentially impact telomere length set point. Loss of RPA1 resets bulk telomeres at a shorter length, with a functional hierarchy for replication group over repair and meiosis group RPA1 subunits. Plants lacking RPA2A, but not RPA2B, harbor short telomeres similar to the replication group. Telomere shortening does not correlate with decreased telomerase activity or deprotection of chromosome ends in rpa mutants. However, in vitro assays show that RPA1B2A3B unfolds telomeric G-quadruplexes known to inhibit replications fork progression. We also found that ATR deficiency can partially rescue short telomeres in rpa2a mutants, although plants exhibit defects in growth and development. Unexpectedly, the telomere shortening phenotype of rpa2a mutants is completely abolished in plants lacking the RTEL1 helicase. RTEL1 has been implicated in a variety of nucleic acid transactions, including suppression of homologous recombination. Thus, the lack of telomere shortening in rpa2a mutants upon RTEL1 deletion suggests that telomere replication defects incurred by loss of RPA may be bypassed by homologous recombination. Taken together, these findings provide new insight into how RPA cooperates with replication and recombination machinery to sustain telomeric DNA.

DNA-mediated coupling of ATPase, translocase and nuclease activities of a Type ISP restriction-modification enzyme.

Enzymes involved in nucleic acid transactions often have a helicase-like ATPase coordinating and driving their functional activities, but our understanding of the mechanistic details of their coordination is limited. For example, DNA cleavage by the antiphage defense system Type ISP restriction-modification enzyme requires convergence of two such enzymes that are actively translocating on DNA powered by Superfamily 2 ATPases. The ATPase is activated when the enzyme recognizes a DNA target sequence. Here, we show that the activation is a two-stage process of partial ATPase stimulation upon recognition of the target sequence by the methyltransferase and the target recognition domains, and complete stimulation that additionally requires the DNA to interact with the ATPase domain. Mutagenesis revealed that a β-hairpin loop and motif V of the ATPase couples DNA translocation to ATP hydrolysis. Deletion of the loop inhibited translocation, while mutation of motif V slowed the rate of translocation. Both the mutations inhibited the double-strand (ds) DNA cleavage activity of the enzyme. However, a translocating motif V mutant cleaved dsDNA on encountering a translocating wild-type enzyme. Based on these results, we conclude that the ATPase-driven translocation not only brings two nucleases spatially close to catalyze dsDNA break, but that the rate of translocation influences dsDNA cleavage.

Adapting TPC-C benchmark to measure performance of multi-document transactions in MongoDB

MongoDB is a popular distributed database that supports replication, horizontal partitioning (sharding), a flexible document schema and ACID guarantees on the document level. While it is generally grouped with "NoSQL" databases, MongoDB provides many features similar to those of traditional RDBMS such as secondary indexes, an ad hoc query language, support for complex aggregations, and new as of version 4.0 multi-statement, multi-document ACID transactions. We looked for a well understood OLTP workload benchmark to use in our own system performance test suite to establish a baseline of transaction performance to enable flagging performance regressions, as well as improvements as we continue to add new functionality. While there exist many published and widely used benchmarks for RDBMS OLTP workloads, there are none specifically for document databases. This paper describes the process of adapting an existing traditional RDBMS benchmark to MongoDB query language and transaction semantics to allow measuring transaction performance. We chose to adapt the TPC-C benchmark even though it assumes a relational database schema and SQL, hence extensive changes had to be made to stay consistent with MongoDB best practices. Our goal did not include creating official TPC-C certifiable results, however, every attempt was made to stay consistent with the spirit of the original benchmark specification as well as to be compliant to all specification requirements where possible. We discovered that following best practices for document schema design achieves better performance than using required normalized schema. All the source code used and validation scripts are published in github to allow the reader to recreate and verify our results.

Schema-Based JSON Data Stores in Relational Databases

JSON is a simple, compact and light weighted data exchange format to communicate between web services and client applications. NoSQL document stores evolve with the popularity of JSON, which can support JSON schema-less storage, reduce cost, and facilitate quick development. However, NoSQL still lacks standard query language and supports eventually consistent BASE transaction model rather than the ACID transaction model. This is very challenging and a burden on the developer. The relational database management systems (RDBMS) support JSON in binary format with SQL functions (also known as SQL/JSON). However, these functions are not standardized yet and vary across vendors along with different limitations and complexities. More importantly, complex searches, partial updates, composite queries, and analyses are cumbersome and time consuming in SQL/JSON compared to standard SQL operations. It is essential to integrate JSON into databases that use standard SQL features, support ACID transactional models, and has the capability of managing and organizing data efficiently. In this article, we empower JSON to use relational databases for analysis and complex queries. The authors reveal that the descriptive nature of the JSON schema can be utilized to create a relational schema for the storage of the JSON document. Then, the powerful SQL features can be used to gain consistency and ACID compatibility for querying JSON instances from the relational schema. This approach will open a gateway to combine the best features of both worlds: the fast development of JSON, consistency of relational model, and efficiency of SQL.

SLOG

For decades, applications deployed on a world-wide scale have been forced to give up at least one of (1) strict serializability (2) low latency writes (3) high transactional throughput. In this paper we discuss SLOG: a system that avoids this tradeoff for workloads which contain physical region locality in data access. SLOG achieves high-throughput, strictly serializable ACID transactions at geo-replicated distance and scale for all transactions submitted across the world, all the while achieving low latency for transactions that initiate from a location close to the home region for data they access. Experiments find that SLOG can reduce latency by more than an order of magnitude relative to state-of-the-art strictly serializable geo-replicated database systems such as Spanner and Calvin, while maintaining high throughput under contention.

Ocean vista

Providing ACID transactions under conflicts across globally distributed data is the Everest of transaction processing protocols. Transaction processing in this scenario is particularly costly due to the high latency of cross-continent network links, which inflates concurrency control and data replication overheads. To mitigate the problem, we introduce Ocean Vista - a novel distributed protocol that guarantees strict serializability . We observe that concurrency control and replication address different aspects of resolving the visibility of transactions, and we address both concerns using a multi-version protocol that tracks visibility using version watermarks and arrives at correct visibility decisions using efficient gossip. Gossiping the watermarks enables asynchronous transaction processing and acknowledging transaction visibility in batches in the concurrency control and replication protocols, which improves efficiency under high cross-datacenter network delays. In particular, Ocean Vista can process conflicting transactions in parallel, and supports efficient write-quorum / read-one access using one round trip in the common case. We demonstrate experimentally in a multi-data-center cloud environment that our design outperforms a leading distributed transaction processing engine (TAPIR) more than 10-fold in terms of peak throughput, albeit at the cost of additional latency for gossip. The latency penalty is generally bounded by one wide area network (WAN) round trip time (RTT), and in the best case (i.e., under light load) our system nearly breaks even with TAPIR by committing transactions in around one WAN RTT.

TxFS

We introduce TxFS, a transactional file system that builds upon a file system’s atomic-update mechanism such as journaling. Though prior work has explored a number of transactional file systems, TxFS has a unique set of properties: a simple API, portability across different hardware, high performance, low complexity (by building on the file-system journal), and full ACID transactions. We port SQLite, OpenLDAP, and Git to use TxFS and experimentally show that TxFS provides strong crash consistency while providing equal or better performance.