Using Cuckoo Filters to Improve Performance in Object Store-based Very Large Databases

Abstract

Today, there is a growing trend to leverage a cheaper data lake such as AWS S3 for storage of tables in very large databases deviating from traditional databases. Here, data is organized as objects (collection of tuples) before placing them in a bucket/container on the external object storage. To allow multiple readers and writers accessing the same objects in parallel, objects are multi-versioned using multiversioned concurrency control (MVCC). Typically, write operations on objects are not done in-place. Instead, newer versions of the modified objects are created. In the context of locking mechanism used today, only table level locks are being placed, which prevents writers on disjoint object sets of the same logical table executing concurrently. In some databases, optimistic concurrency control is used wherein no locks are placed upfront and concurrent writes can proceed. However, these are prone to additional rollbacks placing significant overhead on the system when modifications are a significant percentage of the overall load. In this paper, we focus on pessimistic concurrency control, managing object level locking efficiently using cuckoo filters. To enable transactions to lock only relevant objects of a table, we maintain object-level metadata including information such as minimum, maximum, unique values, number of non-null values, etc., for selected attributes in the table. With such a rich set of information available, objects needed by a transaction for processing could be identified with a greater precision. In summary, we propose an object-level locking mechanism leveraging cuckoo filters to improve the overall throughput of concurrent write transactions with a pessimistic multiversion concurrency control mechanism. Through experimental results and probabilistic analysis, we show the significant improvements (up to 60% in elapsed time and 171% in throughput) using the proposed scheme.