Key-value Store Research Articles

Enabling Encrypted Rich Queries in Distributed Key-Value Stores

To accommodate massive digital data, distributed data stores have become the main solution for cloud services. Among others, key-value stores are widely adopted due to their superior performance. But with the rapid growth of cloud storage, there are growing concerns about data privacy. In this paper, we design and build EncKV, an encrypted and distributed key-value store with rich query support. First, EncKV partitions data records with secondary attributes into a set of encrypted key-value pairs to hide relations between data values. Second, EncKV uses the latest cryptographic techniques for searching on encrypted data, i.e., searchable symmetric encryption (SSE) and order-revealing encryption (ORE) to support secure exact-match and range-match queries, respectively. It further employs a framework for encrypted and distributed indexes supporting query processing in parallel. To address inference attacks on ORE, EncKV is equipped with an enhanced ORE scheme with reduced leakage. For practical considerations, EncKV also enables secure system scaling in a minimally intrusive way. We complete the prototype implementation and deploy it on Amazon Cloud. Experimental results confirm that EncKV preserves the efficiency and scalability of distributed key-value stores.

Block as a value for SQL over NoSQL

This paper presents Zidian, a middleware for key-value (KV) stores to speed up SQL query evaluation over NoSQL. As opposed to common practice that takes a tuple id or primary key as key and the entire tuple as value, Zidian proposes a block-as-a-value model BaaV. BaaV represents a relation as keyed blocks ( k, B ), where k is a key of a block (a set) B of partial tuples. We extend relational algebra to BaaV. We show that under BaaV, Zidian substantially reduces data access and communication cost. We provide characterizations (sufficient and necessary conditions) for (a) result-preserving queries, i.e., queries covered by available BaaV stores, (b) scan-free queries, i.e., queries that can be evaluated without scanning any table, and (c) bounded queries, i.e., queries that can be answered by accessing a bounded amount of data. We show that in parallel processing, Zidian guarantees (a) no scans for scan-free queries, (b) bounded communication cost for bounded queries; and (c) parallel scalability, i.e., speed up when adding processors. Moreover, Zidian can be plugged into existing SQL-over-NoSQL systems and retains horizontal scalability. Using benchmark and real-life data, we empirically verify that Zidian improves existing SQL-over-NoSQL systems by 2 orders of magnitude on average.

The Matrix KV Storage System Based on NVM Devices.

The storage device based on Nonvolatile Memory (NVM devices) has high read/write speed and embedded processor. It is a useful way to improve the efficiency of Key-Value (KV) application. However it still has some limitations such as limited capacity, poorer computing power compared with CPU, and complex I/O system software. Thus it is not an effective way to construct KV storage system with NVM devices directly. We analyze the characteristics of NVM devices and demands of KV application to design the matrix KV storage system based on NVM Devices. The group collaboration management based on Bloomfilter, intragroup optimization based on competition, embedded KV management based on B+-tree, and the new interface of KV storage system are presented. Then, the embedded processor in the NVM device and CPU can be comprehensively utilized to construct a matrix KV pair management system. It can improve the storage and management efficiency of massive KV pairs, and it can also support the efficient execution of KV applications. A prototype is implemented named MKVS (the matrix KV storage system based on NVM devices) to test with YCSB (Yahoo! Cloud System Benchmark) and to compare with the current in-memory KV store. The results show that MKVS can improve the throughput by 5.98 times, and reduce the 99.7% read latency and 77.2% write latency.

Implementing efficient data compression and encryption in a persistent key-value store for HPC

Recently, persistent data structures, like key-value stores (KVSs), which are stored in a high-performance computing (HPC) system’s nonvolatile memory, provide an attractive solution for a number of emerging challenges like limited I/O performance. Data compression and encryption are two well-known techniques for improving several properties of such data-oriented systems. This article investigates how to efficiently integrate data compression and encryption into persistent KVSs for HPC with the ultimate goal of hiding their costs and complexity in terms of performance and ease of use. Our compression technique exploits deep memory hierarchy in an HPC system to achieve both storage reduction and performance improvement. Our encryption technique provides a practical level of security and enables sharing of sensitive data securely in complex scientific workflows with nearly imperceptible cost. We implement the proposed techniques on top of a distributed embedded KVS to evaluate the benefits and costs of incorporating these capabilities along different points in the dataflow path, illustrating differences in effective bandwidth, latency, and additional computational expense on Swiss National Supercomputing Centre’s Grand Tavé and National Energy Research Scientific Computing Center’s Cori.

Pipeline-integrity: Scaling the use of authenticated data structures up to the cloud

Abstract Public cloud storage services are widely adopted for their scalability and low cost. However, delegating the management of the storage has serious implications from the security point of view. We focus on integrity verification of query results based on the use of Authenticated Data Structures (ADS). An ADS enables efficient updates of a cryptographic digest, when data changes, and efficient query verification against this digest. Since, the digest can be updated (and usually signed) exclusively with the intervention of a trusted party, the adoption of this approach is source of a serious performance degradation, in particular when the trusted party is far from the server that stores the ADS. In this paper, we show a protocol for a key–value storage service that provides ADS-enabled integrity-protected queries and updates without impairing scalability, even in the presence of large network latencies between trusted clients and an untrusted server. Our solution complies with the principle of the cloud paradigm in which services should be able to arbitrarily scale with respect to number of clients, requests rates, and data size keeping response time limited. We formally prove that our approach is able to detect server misbehaviour in a setting whose consistency rules are only slightly weaker than those guaranteed by previous results. We provide experimental evidence for the feasibility and scalability of our approach.

MRPrimerW2: an enhanced tool for rapid design of valid high-quality primers with multiple search modes for qPCR experiments.

For the best results in quantitative polymerase chain reaction (qPCR) experiments, it is essential to design high-quality primers considering a multitude of constraints and the purpose of experiments. The constraints include many filtering constraints, homology test on a huge number of off-target sequences, the same constraints for batch design of primers, exon spanning, and avoiding single nucleotide polymorphism (SNP) sites. The target sequences are either in database or given as FASTA sequences, and the experiment is for amplifying either each target sequence with each corresponding primer pairs designed under the same constraints or all target sequences with a single pair of primers. Many websites have been proposed, but none of them including our previous MRPrimerW fulfilled all the above features. Here, we describe the MRPrimerW2, the update version of MRPrimerW, which fulfils all the features by maintaining the advantages of MRPrimerW in terms of the kinds and sizes of databases for valid primers and the number of search modes. To achieve it, we exploited GPU computation and a disk-based key-value store using PCIe SSD. The complete set of 3 509 244 680 valid primers of MRPrimerW2 covers 99% of nine important organisms in an exhaustive manner. Free access: http://MRPrimerW2.com

Fast In-Memory Key–Value Cache System with RDMA

The quick advances of Cloud and the advent of Fog computing impose more and more critical demand for computing and data transfer of low latency onto the underlying distributed computing infrastructure. Remote direct memory access (RDMA) technology has been widely applied for its low latency of remote data access. However, RDMA gives rise to a host of challenges in accelerating in-memory key–value stores, such as direct remote memory writes, making the remote system more vulnerable. This study presents an in-memory key–value system based on RDMA, named Craftscached, which enables: (1) buffering remote memory writes into a communication cache memory to eliminate direct remote memory writes to the data memory area; (2) dividing the communication cache memory into RDMA-writable and RDMA-readable memory zones to reduce the possibility of data corruption due to stray memory writes and caching data into an RDMA-readable memory zone to improve the remote memory read performance; and (3) adopting remote out-of-place direct memory write to achieve high performance of remote read and write. Experimental results in comparison with Memcached indicate that Craftscached provides a far better performance: (1) in the case of read-intensive workloads, the data access of Craftscached is about 7–43[Formula: see text] and 18–72.4% better than those of TCP/IP-based and RDMA-based Memcached, respectively; (2) the memory utilization of small objects is more efficient with only about 3.8% memory compaction overhead.

Collaborative Compaction Optimization System using Near-Data Processing for LSM-tree-based Key-Value Stores

Log-structured merge tree (LSM-tree) based key–value stores are widely employed in large-scale storage systems. In compaction, high-level sorted string table files (i.e., SSTables) are merged with low-level overlapping key ranges and sorted for data queries. However, the compaction process incurs write amplification, which degrades system performance particularly under update-intensive workloads. Current optimizations mostly focus on reducing the overload of compaction in the host but rarely exploit parallelisms between the host and the device to fully utilize computing resources in the entire system for optimizing compaction performance. In this study, we propose Co-KV, a Collaborative Key-Value store, to improve compaction performance in LSM-tree-based key–value stores. Co-KV is based on a near-data processing (i.e., NDP) model-enabled storage device. Co-KV exhibits the following advantages: (1) it reduces write amplification and host-side CPU costs using a compaction offloading scheduling between a host computer and an NDP-enabled storage device; (2) it relieves the overload associated with data transfer between the host and the storage device; and (3) it improves the compaction of LSM-tree based key–value stores under update-intensive workloads.We employed an Open Ethernet Driver (OED), which is a real-world NDP platform as the testbed for our experiments. Extensive db_bench evaluations demonstrate that Co-KV achieves overall throughput improvements of approximately 1.75x, CPU cost reductions of approximately 68.1%, and write amplification reductions by up to 50.0% over the state-of-the-art LevelDB. Under YCSB workloads, Co-KV increases throughput by 1.7x ∼ 1.9x while decreasing the write amplification and average latency by up to 50.0% and 46.3%, respectively.

Challenges in NoSQL-Based Distributed Data Storage: A Systematic Literature Review

Key-Value stores (KVSs) are the most flexible and simplest model of NoSQL databases, which have become highly popular over the last few years due to their salient features such as availability, portability, reliability, and low operational cost. From the perspective of software engineering, the chief obstacle for KVSs is to achieve software quality attributes (consistency, throughput, latency, security, performance, load balancing, and query processing) to ensure quality. The presented research is a Systematic Literature Review (SLR) to find the state-of-the-art research in the KVS domain, and through doing so determine the major challenges and solutions. This work reviews the 45 papers between 2010–2018 that were found to be closely relevant to our study area. The results show that performance is addressed in 31% of the studies, consistency is addressed in 20% of the studies, latency and throughput are addressed in 16% of the studies, query processing is addressed in 13% of studies, security is addressed in 11% of the studies, and load balancing is addressed in 9% of the studies. Different models are used for execution. The indexing technique was used in 20% of the studies, the hashing technique was used in 13% of the studies, the caching and security techniques were used together in 9% of the studies, the batching technique was used in 5% of the studies, the encoding techniques and Paxos technique were used together in 4% of the studies, and 36% of the studies used other techniques. This systematic review will enable researchers to design key-value stores as efficient storage. Regarding future collaborations, trust and privacy are the quality attributes that can be addressed; KVS is an emerging facet due to its widespread popularity, opening the way to deploy it with proper protection.

Prospect of Introducing a New Solution - MongoDB

There are many awesome features of MongoDB that have made MongoDB so perspective. Document-oriented databases share a lot in common with relational databases. Therefore, rather than tiptoeing around it, lets simply state that MongoDB should be seen as a direct alternative to relational databases. Where one might see Lucene as enhancing a relational database with full text indexing, or Redis as a persistent key-value store, MongoDB is a central repository for our data. I didnt call MongoDB a replacement for relational databases, but rather an alternative. Its a tool that can do what a lot of other tools can do. Some of it MongoDB does better, some of it MongoDB does worse. MongoDB might fit into your existing system. There are enough new and competing storage technologies that its easy to get overwhelmed by all of the choices.

Understanding and improvement of the selection of replica servers in key–value stores

In current large-scale distributed key–value stores, the tail latency of the hundreds of key–value access operations generated by an end-user request determines the response time of this request. Therefore, this tail latency has great impacts on the user experience and revenue. Replica selection algorithms, which select the best replica server for the service of each key–value access operation as much as possible, is the key to cut the tail latency of these key–value access operations. This paper summarizes current replica selection algorithms, including both the algorithms employed by current key–value stores and the classic algorithms of other similar systems. These algorithms are classified into three categories: information-agnostic, client-independence and feedback, according to their demanded information. As a step further, simulation-based performance analysis of these algorithms is conducted. The result brings us the insights that the response time (RPT) is useful to measure the service rate, but will lead to the herd behaviors. Moreover, the number of outstanding key–value access operations (OSKs) is helpful to both the selection of the fastest replica server and the avoidance of herd behaviors. Based on these insights, we design the L2 algorithm by assembling the basic ideas of the Least OSK algorithm and the Least RPT algorithm. The L2 algorithm is much simpler than the recently proposed C3 algorithm and has a similar best performance with C3 as confirmed by the simulation and experimental results.

Toward Secure and Scalable Computation in Internet of Things Data Applications

The ever-growing of Internet of Things (IoT) data and the new spectrum of data applications have stimulated IoT clients to outsource their data to cloud servers or datacenters. Apart from storage service, the IoT clients also desires the servers to execute functional operations per client’s request. In this paper, we aim to design the secure mechanisms that allow the IoT clients to outsource their encrypted data to geographically distributed servers while supporting homomorphic computation functions. We leverage the distributed index framework to disassemble and spread data evenly across geographically distributed servers while employing the key–value store as the underlying structure for fast data retrieval. To support computing over encrypted data, we customize Shamir’s secret sharing into our mechanisms to design a tunable scheme for the adaption of different IoT application scenarios. In particular, we design three tunable protocols to achieve the effective additive homomorphic computations while approaching efficiency in terms of servers utilization, computation, and storage overhead. Even the designs focus on the additive computation, we show that it can be readily extended to other types of homomorphic computations as well as verifying the correctness of stored data. Based on the proposed protocols, we design system prototypes, deploy them in Amazon Web services, and evaluate our construction experimentally. Through experimental results, we show that our designs can achieve the efficiency in various perspectives.

IMPLEMENTING BI-TEMPORAL PROPERTIES INTO VARIOUS NOSQL DATABASE CATEGORIES

NoSQL database systems have emerged and developed at an accelerating rate in the last years. Attractive properties such as scalability and performance, which are needed by many applications today, contributed to their increasing popularity. Time is very important aspect in many applications. Many NoSQL database systems do not offer built in management for temporal properties. In this paper, we discuss how we can embed temporal properties in NoSQL databases. We review and differentiate between the most popular NoSQL stores. Moreover, we propose various solutions to modify data models for embedding bitemporal properties in two of the most popular categories of NoSQL databases (Key-value stores and Column stores). In addition, we give examples of how to represent bitemporal properties using Redis Key-value store and Cassandra column oriented store. This work can be used as basis for designing and implementing temporal operators and temporal data management in NoSQL databases.

KVEFS: Encrypted File System Based on Distributed Key-Value Stores and FUSE

File System is an important component of a secure operating system. The need to build data protection systems is extremely important in open source operating systems, high mobility hardware systems, and miniaturization of storage devices that make systems available. It is clear that the value of the data is much larger than the value of the storage device. Computers access protection mechanism does not work if the thief retrieves the hard drive from the computer and reads data from it on another computer. Encrypted File System (EFS) is a secure level of operating system kernel. EFS uses cryptography to encrypt or decrypt files and folders when they are being saved or retrieved from a hard disk. EFS is often integrated transparently in operating system There are many encrypted filesystems commonly used in Linux operating systems. However, they have some limitations, which are the inability to hide the structure of the file system. This is a shortcoming targeted by the attacker, who will try to decrypt a file to find the key and then decrypt the entire file system. In this paper, we propose a new architecture of EFS called KVEFS which is based on cryptographic algorithms, FUSE library and key-value store. Our method makes EFS portable and flexible; Kernel size will not increase in Operating System.

범위 기반 로그 구조 병합 트리를 활용한 키-값 저장소의 최적화 방법

범위 기반 로그 구조 병합 트리를 활용한 키-값 저장소의 최적화 방법

Haste makes waste: The On–Off algorithm for replica selection in key–value stores

In current large-scale distributed key–value stores, the tail latency of the key–value accesses generated by end-user requests is crucial to the response time of these requests. To cut the tail latency, the replica selection algorithm, which selects a replica server for each key by a client and thus determines the latency of each key–value access, is crucial. Naturally, all of current replica selection algorithms send keys out immediately to reduce the tail latencies of key–value accesses. In this paper, we find that sending out keys in haste makes waste of the chance to select a better replica server certain time later, and suggest to await for a better replica server available when all current replica servers are bad. To realize this idea, we develop the On–Off algorithm, which recognizes bad replica servers according to the feedback information and put them into the OFF state. Special attention is paid on the time interval replica servers stay at the OFF state. Obviously, the On–Off algorithm adds awaiting time at clients, but can greatly reduce the dominated queuing delays at replica servers. In total, the On–Off algorithm improves the 99th latency by about 29% under the default simulation configuration and outperforms the C3 algorithm proposed recently under kinds of scenarios.

TeksDB

In this paper, we examine the design tradeoffs of existing in-memory data structures of a state-of-the-art key-value store. We observe that no data structures provide both fast point-accesses and consistent ranged- retrievals, and naive amalgamations of existing structures fail to get the best of both worlds. Furthermore, our experiments reveal a performance anomaly when increasing the memory size: as more key-value pairs are maintained in memory, the shortcomings of the data structures exacerbate. To address the above problems, we present TeksDB, a fast and consistent key-value store with a novel in-memory data structure, which effciently handles both point- and ranged- accesses at a modest increase in memory footprint. Our evaluation demonstrates that TeksDB outperforms RocksDB by 3.6×, 9×, and 4.5× for get, scan, and range_query, respectively. The effectiveness of TeksDB extends to real-world workloads, achieving up to 3.3× speedup for YCSB.

Design and implementation of skiplist-based key-value store on non-volatile memory

Non-volatile random access memory (NVRAM) is a promising approach to persistent data storage with outstanding advantages over traditional storage devices, such as hard disk drives (HDDs) and solid state drives (SSDs). Some of its biggest advantages are its DRAM-like read latency and microsecond-level write latency, which are several hundred times faster than those in the original block device. However, one of the issues with using NVRAM as a storage device is designing an indexing system for its data stores to fully utilize NVRAM characteristics. The state-of-the-art indexing systems of non-volatile key-value stores are usually based on B+-trees or their variants, which were originally designed for block-based storage devices with better sequential performance than random performance. The semantics of B+-tree require data being sorted into leaf nodes and inner nodes and frequent splitting and merging to keep balanced. However, all the sorting, splitting, and merging operations cause extra write to NVRAM, which decreases its performance. In this article, we propose NV-Skiplist, a skiplist-based indexing system for key-value stores on NVRAM that fully uses the features of both NVRAM and DRAM. NV-Skiplist constructs its bottom layer in non-volatile memory to maintain data persistence and support range scans. It builds its upper layers in DRAM to retain rapid index searching and prevent consistently large overhead. We also propose a multiranged variant of NV-Skiplist to increase its search performance and scalability. We evaluate the performance of NV-Skiplist and wB+-tree which is a state-of-art scheme on an NVRAM emulator on a server with an Intel Xeon E5-2620 v2 processor. The results show that our design outperforms the original tree-based, non-volatile key-value stores up to 48%.

Leveraging Glocality for Fast Failure Recovery in Distributed RAM Storage

Distributed RAM storage aggregates the RAM of servers in data center networks (DCN) to provide extremely high I/O performance for large-scale cloud systems. For quick recovery of storage server failures, MemCube [53] exploits the proximity of the BCube network to limit the recovery traffic to the recovery servers’ 1-hop neighborhood. However, the previous design is applicable only to the symmetric BCube( n , k ) network with n k +1 nodes and has suboptimal recovery performance due to congestion and contention. To address these problems, in this article, we propose CubeX, which (i) generalizes the “1-hop” principle of MemCube for arbitrary cube-based networks and (ii) improves the throughput and recovery performance of RAM-based key-value (KV) store via cross-layer optimizations. At the core of CubeX is to leverage the glocality (= globality + locality) of cube-based networks: It scatters backup data across a large number of disks globally distributed throughout the cube and restricts all recovery traffic within the small local range of each server node. Our evaluation shows that CubeX not only efficiently supports RAM-based KV store for cube-based networks but also significantly outperforms MemCube and RAMCloud in both throughput and recovery time.

LSM-Tree Managed Storage for Large-Scale Key-Value Store

Key-value stores are increasingly adopting LSM-trees as their enabling data structure in the backend block storage, and persisting their clustered data through a block manager, usually a file system. In general, a file system is expected to not only provide file/directory abstraction to organize data but also retain the key benefits of LSM-trees, namely, sequential and aggregated I/O patterns on the physical device. Unfortunately, our in-depth experimental analysis reveals that some of these benefits of LSM-trees can be completely negated by the underlying file level indexes from the perspectives of both data layout and I/O processing. As a result, the write performance of LSM-trees is kept at a level far below that promised by the sequential bandwidth offered by the storage devices. In this paper, we address this problem and propose LDS, an LSM-tree Direct Storage system that manages the storage space based on the LSM-tree objects and provides simplified consistency control by leveraging the copy-on-write nature of the LSM-tree structure, to fully reap the benefits of LSM-trees. Running LevelDB, a popular LSM-tree based key-value store, on LDS as a baseline, comparing that to LevelDB running on three representative file systems (ext4, f2fs, btrfs) with HDDs and SSDs, respectively, we evaluate and study the performance potentials of LSM-trees. Evaluation results show that the write throughputs of LevelDB can be improved by from 1.8× to 3× on HDDs, and from 1.3× to 2.5× on SSDs, by employing the LSM-tree friendly data layout of LDS.