Transactional Workloads Research Articles

Optimizing Cloud Data Storage: Evaluating File Formats for Efficient Data Warehousing

This paper presents a detailed analysis of three widely-used data storage formats—Parquet, Avro, and ORC— evaluating their performance across key metrics such as query execution, compression efficiency, data skipping, schema evolution, and throughput. Each format offers distinct advantages depending on the nature of the workload. Parquet is optimized for read-heavy analytical queries, providing excellent compression and efficient query performance through its columnar structure. Avro excels in write-heavy, real-time data streaming scenarios, where schema flexibility and backward compatibility are crucial. ORC balances the two, offering strong support for analytical and transactional workloads, especially in handling complex queries and nested data structures. This comparative study highlights the contexts in which each format performs best, providing valuable insights into the trade-offs associated with their use in cloud data warehouses and large-scale data processing environments.

Horizontally Scalable Implementation of a Distributed DBMS Delivering Causal Consistency via the Actor Model

Causal Consistency has been proven to be the strongest type of consistency that can be achieved in a fault-tolerant, distributed system. This paper describes an implementation of D-Thespis, which is an approach that employs the actor mathematical model of concurrent computation to establish a distributed middleware that enforces causal consistency on a widely used relational database management system (RDBMS). D-Thespis prioritises developer experience by encapsulating the intricacies of causal consistency behind an interface that is accessible over standard REST protocol. Here, we discuss several novel results. Firstly, we define a method that builds a causally consistent DBMS supporting elastic horizontal scalability. Secondly, we deliver a cloud-native implementation of the middleware and provide results and insights on 6804 benchmark configurations executed on our implementation while running on a public cloud infrastructure across several data centres. The evaluation concerns transaction processing performance, an evaluation of our implementation’s update visibility latency, and a memory profiling exercise. The results of our evaluation show that under a transactional workload, a single-node installation of our implementation of D-Thespis is 1.5 times faster than a relational DBMS running serialisable transaction processing, while the performance of the middleware can improve by more than three times when scaled horizontally within the same data centre. Our study of the memory profile of the D-Thespis implementation shows that the system distributes its memory requirements evenly across all the available machines, as it is scaled horizontally. Finally, we also illustrate how our middleware propagates data changes across geographically-distributed infrastructures in a timely manner: our tests show that most of the effects of data change operations in one data centre are available in a remote data centre within less than 300 ms over and above the network round trip latency between the two data centres.

On the scalability of the speedup considering the overhead of consolidating virtual machines in servers for data centers

Virtualization technologies are extensively utilized in data centers, particularly cloud computing. This facilitates data center management and diminishes the number of physical machines (servers) and, subsequently, their cooling requirements, leading to cost, space, and power consumption reductions. When applications in data centers are executing independent parallel transactions, but with similar performance requirements, the appropriate level of virtual machine consolidation on a server poses a fundamental challenge for capacity planning. This article introduces a method to evaluate the performance speedup achieved through virtualization on any server and the effects of virtualization and consolidation overheads on physical or virtual machine scalability. This research formalizes the speedup and overheads, using classical computer architecture statements. but at the same time proposes a new method to analyze these overhead amounts and types, showing the scalability and efficiency of different consolidations in the same server and its comparison against no consolidation. This work also proposes a new way to determine the optimal number of physical servers and the optimal number of consolidated virtual machines for a given transaction workload. The real experimentation was performed with different workload sizes, types of virtualizations and different servers. The method presented also facilitates the representation of linear scalability against the real degree of parallelism of either physical machines or consolidated virtual machines for a given transaction workload, as well as striking the right balance between speedup and energy in virtual server consolidation.

Integrating systems of systems with a federation of rule engines

Systems of Systems (SoSs) integrate many critical systems our society relies on. In designing individual systems, stakeholders use bespoke protocols, custom information models, and proprietary components with limited computational resources from various vendors. We present a reference architecture that allows multiple stakeholders to carry out a flexible integration without giving up control to a single entity in the presence of the aforementioned limitations. Our architecture relies on rule engines and graph data model to integrated systems flexibly even when black-box components are used. At the same time, a federation of the rule engines allows each stakeholder to retain control over the rules that reflect their policies. We also rely on a common information model based on ontologies to account for the information model mismatch and reduce the duplication of integration efforts. Moving rule execution to the standalone rule engines allows deployment in resource-constrained and proprietary environments. A uniform application programming interface (API) is used to integrate rule engines across systems as well as components within each system with a respective rule engine. We also present a novel algorithm to determine dependencies across rules deployed in different rule engines within the federation. This allows domain experts to develop rules as usual without having to deal with the distributed aspect of the system. We also present a proof of the sufficient condition to ensure all necessary notifications will be sent to ensure correct rule activation across different rule engines. Compared to other systems involving distributed rules, the proposed architecture is well-suited for the integration of transactional workloads commonly found in enterprises. The qualitative evaluation based on the Architecture Tradeoff Analysis Method (ATAM), applied to a telecommunications use case, shows that the architecture possesses the “interoperability”, “modifiability”, and the “functional completeness” quality attributes with a trade-off around rule expressiveness. The quantitative evaluation demonstrates speedup over the single-node setup in most scenarios except in case of highly optimized rules and a poor network performance simultaneously (tr<10ms, tn=100ms).

Robustness against Read Committed for Transaction Templates with Functional Constraints

The popular isolation level Multiversion Read Committed (RC) trades some of the strong guarantees of serializability for increased transaction throughput. Sometimes, transaction workloads can be safely executed under RC obtaining serializability at the lower cost of RC. Such workloads are said to be robust against RC. Previous work has yielded a tractable procedure for deciding robustness against RC for workloads generated by transaction programs modeled as transaction templates. An important insight of that work is that, by more accurately modeling transaction programs, we are able to recognize larger sets of workloads as robust. In this work, we increase the modeling power of transaction templates by extending them with functional constraints, which are useful for capturing data dependencies like foreign keys. We show that the incorporation of functional constraints can identify more workloads as robust that otherwise would not be. Even though we establish that the robustness problem becomes undecidable in its most general form, we show that various restrictions on functional constraints lead to decidable and even tractable fragments that can be used to model and test for robustness against RC for realistic scenarios.

On the interactions between ILP and TLP with hardware transactional memory

Hardware implementations of Transactional Memory (HTM) are designed to facilitate efficient thread synchronization in parallel programs, encouraging the use of larger critical sections. By employing optimistic concurrency control to execute transactions speculatively, HTM systems promise to deliver the performance benefits typically associated with fine-grained locks. In doing so, HTM systems must deal with transaction aborts. While under certain conditions aborts may be caused by the inherent limitations of hardware structures employed to implement TM (e.g., caches), conflicting concurrent accesses to shared memory locations are generally the prevailing cause for squashing the work done by a transactionIn this study, we present what we believe to be, to the best of our knowledge, the first characterization of how the aggressiveness of processor cores, particularly their ability to exploit instruction-level parallelism (ILP), interacts with the support for optimistic thread-level speculation offered by HTM systems. We have observed that by adjusting the size of structures that facilitate out-of-order and speculative execution, the number of aborts in the execution of transactional workloads can be altered in best-effort HTM implementations. Our findings indicate that in scenarios with high contention, a smaller number of powerful cores is more suitable, whereas in low contention scenarios, using a larger number of less aggressive cores is preferable. In addition, HTM systems that employ lazy detection and those employing eager detection with requester-stalls resolution, benefit from using simpler cores. In conclusion, abort ratios can be reduced with a careful choice of both processor aggressiveness and design aspects for each application depending on its contention.

When is it safe to run a transactional workload under Read Committed?

The popular isolation level multiversion Read Committed (RC) exchanges some of the strong guarantees of serializability for increased transaction throughput. Nevertheless, transaction workloads can sometimes be executed under RC while still guaranteeing serializability at a reduced cost. Such workloads are said to be robust against RC. This paper provides a high level overview of deciding robustness against RC. In particular, we discuss how a sound and complete test can be obtained through the formalization of transaction templates. We then increase the modeling power of transaction templates by extending them with functional constraints which are useful for capturing data dependencies like foreign keys. We show that the incorporation of functional constraints can identify more workloads as robust than would otherwise be the case. Even though the robustness problem becomes undecidable in its most general form, we establish that various restrictions on functional constraints lead to decidable and even tractable results that can be used to model and test for robustness against RC for practical scenarios.

Technical Perspective: When is it safe to run a transactional workload under Read Committed?

A data management platform provides many capabilities to assist the data owner, application coder, or end-user. For example, it should support an expressive query language, schema definition, and sophisticated access control. Another way many platforms add value is through a transaction mechanism, which allows the application programmer to indicate that a stretch of code, including multiple accesses to data, represents a single real-world activity and so all these steps should happen as if a single step, despite really being interleaved with other programs, or perhaps cancelled after partial execution. If the platform perfectly hides interleaving of different activities, the execution is called serializable, and this is a great aid to protecting data quality. Any integrity constraint over the data (whether explicitly declared in schema or not) which is preserved by each transaction running alone, is also valid at the end of any serializable execution of several transactions.

On the Effects of Transaction Data Access Patterns on Performance in Lock-Based Concurrency Control

Transaction Processing (TP) plays a primary role in the design and implementation of IT applications and services. Many TP systems exploit lock-based concurrency control to guarantee atomicity and isolation of transactions that access shared data. In this article, we show that transaction data access patterns, in particular the order of data accesses along the transaction execution, have a noticeable impact on how lock-based concurrency control affects performance. We show that the performance can remarkably change depending on whether transactions, or a percentage of them, access data items following some common ordering rule or not. We investigate on this aspect and its root causes through an analytical modeling approach, and with the evidence of data gathered through both simulation and the execution of real transactional workloads. Finally, we show how the findings of our study can be easily exploited for improving the performance of common transactional workloads.

Transaction Scheduling: From Conflicts to Runtime Conflicts

This paper studies how to improve the performance of main memory multicore OLTP systems for executing transactions with conflicts. A promising approach is to partition transaction workloads into mutually conflict-free clusters, and distribute the clusters to different cores for concurrent execution. We show that if transactions in each cluster are properly scheduled, transactions that are traditionally considered conflicting can be executed without conflicts at runtime. In light of this, we propose to schedule transactions and reduce runtime conflicts, instead of partitioning based on the conventional notion of conflicts. We formulate the transaction scheduling problem to minimize runtime conflicts, and show that the problem is NP-complete. This said, we develop an efficient scheduling algorithm to improve parallelism. Moreover, for transactions that are not packed in batches, we show that runtime conflict analysis also helps reduce conflict penalties, by proposing a proactive deferring method. Using standard and enhanced benchmarks, we show that on average our scheduling and proactive deferring methods improve the throughput of existing partitioners and concurrency control protocols by 131% and 109%, respectively, up to 294% and 152%.

AdaChain: A Learned Adaptive Blockchain

This paper presents AdaChain , a learning-based blockchain framework that adaptively chooses the best permissioned blockchain architecture to optimize effective throughput for dynamic transaction workloads. AdaChain addresses the challenge in Blockchain-as-a-Service (BaaS) environments, where a large variety of possible smart contracts are deployed with different workload characteristics. AdaChain supports automatically adapting to an underlying, dynamically changing workload through the use of reinforcement learning. When a promising architecture is identified, AdaChain switches from the current architecture to the promising one at runtime in a secure and correct manner. Experimentally, we show that AdaChain can converge quickly to optimal architectures under changing workloads and significantly outperform fixed architectures in terms of the number of successfully committed transactions, all while incurring low additional overhead.

Memory-optimized multi-version concurrency control for disk-based database systems

Pure in-memory database systems offer outstanding performance but degrade heavily if the working set does not fit into DRAM, which is problematic in view of declining main memory growth rates. In contrast, recently proposed memory-optimized disk-based systems such as Umbra leverage large in-memory buffers for query processing but rely on fast solid-state disks for persistent storage. They offer near in-memory performance while the working set is cached, and scale gracefully to arbitrarily large data sets far beyond main memory capacity. Past research has shown that this architecture is indeed feasible for read-heavy analytical workloads. We continue this line of work in the following paper, and present a novel multi-version concurrency control approach that enables a memory-optimized disk-based system to achieve excellent performance on transactional workloads as well. Our approach exploits that the vast majority of versioning information can be maintained entirely in-memory without ever being persisted to stable storage, which minimizes the overhead of concurrency control. Large write transactions for which this is not possible are extremely rare, and handled transparently by a lightweight fallback mechanism. Our experiments show that the proposed approach achieves transaction throughput up to an order of magnitude higher than competing disk-based systems, confirming its viability in a real-world setting.

Cost modelling for optimal data placement in heterogeneous main memory

The cost of DRAM contributes significantly to the operating costs of in-memory database management systems (IMDBMS). Persistent memory (PMEM) is an alternative type of byte-addressable memory that offers --- in addition to persistence --- higher capacities than DRAM at a lower price with the disadvantage of increased latencies and reduced bandwidth. This paper evaluates PMEM as a cheaper alternative to DRAM for storing table base data, which can make up a significant fraction of an IMDBMS' total memory footprint. Using a prototype implementation in the SAP HANA IMDBMS, we find that placing all table data in PMEM can reduce query performance in analytical benchmarks by more than a factor of two, while transactional workloads are less affected. To quantify the performance impact of placing individual data structures in PMEM, we propose a cost model based on a lightweight workload characterization. Using this model, we show how to place data pareto-optimally in the heterogeneous memory. Our evaluation demonstrates the accuracy of the model and shows that it is possible to place more than 75% of table data in PMEM while keeping performance within 10% of the DRAM baseline for two analytical benchmarks.

Hihooi: A Database Replication Middleware for Scaling Transactional Databases Consistently

With the advent of the Internet and Internet-connected devices, modern business applications can experience rapid increases as well as variability in transactional workloads. Database replication has been employed to scale performance and improve availability of relational databases but past approaches have suffered from various issues including limited scalability, performance versus consistency tradeoffs, and requirements for database or application modifications. This paper presents Hihooi, a replication-based middleware system that is able to achieve workload scalability, strong consistency guarantees, and elasticity for existing transactional databases at a low cost. A novel replication algorithm enables Hihooi to propagate database modifications asynchronously to all replicas at high speeds, while ensuring that all replicas are consistent. At the same time, a fine-grained routing algorithm is used to load balance incoming transactions to available replicas in a consistent way. Our thorough experimental evaluation with several well-established benchmarks shows how Hihooi is able to achieve almost linear workload scalability for transactional databases.

Transparent Data Transformation

A big dichotomy in data system design is the columnvs. row-stores one. The first supports analytical, and the latter transactional workloads. There have been several efforts to bridge the two, especially in light of new hybrid transactional analytical processing workloads.

Robustness against read committed for transaction templates

The isolation level Multiversion Read Committed (RC), offered by many database systems, is known to trade consistency for increased transaction throughput. Sometimes, transaction workloads can be safely executed under RC obtaining the perfect isolation of serializability at the lower cost of RC. To identify such cases, we introduce an expressive model of transaction programs to better reason about the serializability of transactional workloads. We develop tractable algorithms to decide whether any possible schedule of a workload executed under RC is serializable (referred to as the robustness problem). Our approach yields robust subsets that are larger than those identified by previous methods. We provide experimental evidence that workloads that are robust against RC can be evaluated faster under RC compared to stronger isolation levels. We discuss techniques for making workloads robust against RC by promoting selective read operations to updates. Depending on the scenario, the performance improvements can be considerable. Robustness testing and safely executing transactions under the lower isolation level RC can therefore provide a direct way to increase transaction throughput without changing DBMS internals.

OurRocks: Offloading Disk Scan Directly to GPU in Write-Optimized Database System

The log structured merge (LSM) tree has been widely adopted by database systems owing to its superior write performance. However, LSM-tree based databases face vulnerabilities when processing analytical queries due to the read amplification caused by its architecture and the limited use of storage devices with high bandwidth. To flexibly handle transactional and analytical workloads, we proposed and implemented OurRocks taking full advantage of NVMe SSD and GPU devices, which improves scan performance. Although the NVMe SSD serves multi GB/s I/O rates, it is necessary to solve the data transfer overhead which limits the benefits of the GPU processing. The primary idea is to offload the scan operation to the GPU with filtering predicate pushdown and resolve the bottleneck from the data transfer between devices with direct memory access (DMA). OurRocks benefits from all the features of write-optimized database systems, in addition to accelerating the analytic queries using the aforementioned idea. Experimental results indicate that OurRocks effectively leverages resources of the NVMe SSD and GPU and significantly improves the execution of queries in the YCSB and TPC-H benchmarks, compared to the conventional write-optimized database. Our research demonstrates that the proposed approach can speed up the handling of the data-intensive workloads.

TiDB

Hybrid Transactional and Analytical Processing (HTAP) databases require processing transactional and analytical queries in isolation to remove the interference between them. To achieve this, it is necessary to maintain different replicas of data specified for the two types of queries. However, it is challenging to provide a consistent view for distributed replicas within a storage system, where analytical requests can efficiently read consistent and fresh data from transactional workloads at scale and with high availability. To meet this challenge, we propose extending replicated state machine-based consensus algorithms to provide consistent replicas for HTAP workloads. Based on this novel idea, we present a Raft-based HTAP database: TiDB. In the database, we design a multi-Raft storage system which consists of a row store and a column store. The row store is built based on the Raft algorithm. It is scalable to materialize updates from transactional requests with high availability. In particular, it asynchronously replicates Raft logs to learners which transform row format to column format for tuples, forming a real-time updatable column store. This column store allows analytical queries to efficiently read fresh and consistent data with strong isolation from transactions on the row store. Based on this storage system, we build an SQL engine to process large-scale distributed transactions and expensive analytical queries. The SQL engine optimally accesses row-format and column-format replicas of data. We also include a powerful analysis engine, TiSpark, to help TiDB connect to the Hadoop ecosystem. Comprehensive experiments show that TiDB achieves isolated high performance under CH-benCHmark, a benchmark focusing on HTAP workloads.

Ensuring Consistent Transactions in a Web Service Environment With Prediction-Based Performance Metrics

In this paper, we propose transaction scheduling for web service database transactions. Our solution ensures consistency while preserving efficiency. We propose a prediction-based metric that promotes transactions with reliable reputations based on the transactions performance metrics. Performance metrics are based on the transactions likelihood to commit and their execution time. We propose a customized lock management solution to guarantee execution consistency in concurrent web service environments. We formally prove that our solution guarantees consistent execution history of concurrent web transactions and increases concurrency and performance over traditional locking methods. We developed a simulation using a multi-threaded approach. We generated sample workloads of simulated concurrent transactions over seven tests. Our results show that the solution works comparatively with traditional locking and no-locking solutions with the added benefit of ensured consistency in some cases and deadlock avoidance in others.

Benchmarking on intensive transaction processing

Benchmarks play a crucial role in database performance evaluation, and have been effectively promoting the development of database management systems. With critical transaction processing requirements of new applications, we see an explosion of innovative database technologies for dealing with highly intensive transaction workloads (OLTP) with the obvious characteristics of sharp dynamics, terrificskewness, high contention, or high concurrency (abbr. DSC2), which can not be well described or evaluated by current standard benchmarks. In this paper, based on the representative SecKill applications, we define a pacakge of workloads simulating intensive transactional processing requirements. And we create a general and flexible benchmark framework PeakBench for evaluating intensive OLTP workloads on databases. We are the first work to have full control on simulating DSC2, especially for the fine granularity control for contention generation. With a comprehensive set of experiments conducted on popular open sourced DBMSs compared with the other representative OLTP benchmarks, we completely demonstrate the usefulness of PeakBench.