Yahoo! Cloud Serving Benchmark Research Articles

Offloaded Data Processing Energy Efficiency Evaluation

The growing popularity of mobile and cloud computing raises new challenges related to energy efficiency. This work evaluates four various SQL and NoSQL database solutions in terms of energy efficiency. Namely, Cassandra, MongoDB, Redis, and MySQL are taken into consideration. This study measures energy efficiency of the chosen data storage solutions on a selected set of physical and virtual computing nodes by leveraging Intel RAPL (Running Average Power Limit) technology. Various database usage scenarios are considered in this evaluation including both local usage and remote offloading. Different workloads are benchmarked through the use of YCSB (Yahoo! Cloud Serving Benchmark) tool. Extensive experimental results show that (i) Redis and MongoDB are more efficient in energy consumption under most usage scenarios, (ii) remote offloading saves energy if the network latency is low and destination CPU is significantly more powerful, and (iii) computationally weaker CPUs may sometimes demonstrate higher energy efficiency in terms of J/ops. An energy efficiency measurement framework is proposed in order to evaluate and compare different database solutions based on the obtained experimental results.

Waffle: An Online Oblivious Datastore for Protecting Data Access Patterns

We present Waffle, a datastore that protects an application's data access patterns from a passive persistent adversary. Waffle achieves this without prior knowledge of the input data access distribution, making it the first of its kind to adaptively handle input sequences under a passive persistent adversary. Waffle maintains a constant bandwidth and client-side storage overhead, which can be adjusted to suit the application owner's preferences. This flexibility allows the owner to fine-tune system parameters and strike a balance between security and performance. Our evaluation, utilizing the Yahoo! Cloud Serving Benchmark (YCSB) benchmark and Redis as the backend storage, demonstrates promising results. The insecure baseline outperforms Waffle by a mere 5-6x, whereas Waffle outperforms Pancake-a state-of-the-art oblivious datastore under passive persistent adversaries-by 45-57%, and a concurrent ORAM system, TaoStore, by 102x.

BP-Tree: Overcoming the Point-Range Operation Tradeoff for In-Memory B-Trees

B-trees are the go-to data structure for in-memory indexes in databases and storage systems. B-trees support both point operations (i.e., inserts and finds) and range operations (i.e., iterators and maps). However, there is an inherent tradeoff between point and range operations since the optimal node size for point operations is much smaller than the optimal node size for range operations. Existing implementations use a relatively small node size to achieve fast point operations at the cost of range operation throughput. We present the BP-tree , a variant of the B-tree, that overcomes the decades-old point-range operation tradeoff in traditional B-trees. In the BP-tree, the leaf nodes are much larger in size than the internal nodes to support faster range scans. To avoid any slowdown in point operations due to large leaf nodes, we introduce a new insert-optimized array called the buffered partitioned array (BPA) to efficiently organize data in leaf nodes. The BPA supports fast insertions by delaying ordering the keys in the array. This results in much faster range operations and faster point operations at the same time in the BP-tree. Our experiments show that on 48 hyperthreads, on workloads generated from the Yahoo! Cloud Serving Benchmark (YCSB), the BP-tree supports similar or faster point operation throughput (between .94×-1.2× faster) compared to Masstree and OpenBw-tree, two state-of-the-art in-memory key-value (KV) stores. On a YCSB workload with short scans, the BP-tree is about 7.4× faster than Masstree and 1.6× faster than OpenBw-tree. Furthermore, we extend the YCSB to add large range workloads, commonly found in database applications, and show that the BP-tree is 30× faster than Masstree and 2.5× faster than OpenBw-tree. We also provide a reference implementation for a concurrent B + -tree and find that the BP-tree supports faster (between 1.03×-1.2× faster) point operations when compared to the best-case configuration for B + -trees for point operations while supporting similar performance (about .95× as fast) on short range operations and faster (about 1.3× faster) long range operations.

A Prediction Based Replica Selection Strategy for Reducing Tail Latency in Geo-Distributed Systems

The deployment of modern applications in geo-distributed systems results in performance fluctuation which is a consequence of long-tail latency. To deliver high-quality services these applications always strive to adapt to the changing situation and an appropriate replica selection strategy is one efficient way to achieve this. Several replica selection strategies have already been developed but none of them are efficient enough to reduce tail latency and adapt to the dynamic environment of the geo-distributed systems. In this paper, we present the design and implementation of a prediction based replica selection strategy for reducing tail latency in geo-distributed systems. We have meticulously designed the proposed strategy to adapt to the dynamic behavior of the distributed system. For evaluating its effectiveness in reducing tail latency and adapting to the dynamic behavior of the geo-distributed systems we perform some extensive experiments in a 15 nodes Cassandra cluster that is deployed on Amazon EC2 over 5 geographical regions. For generating test datasets and workloads we use industry-standard Yahoo Cloud Serving Benchmark (YCSB). Our experimental results show that the proposed strategy not only reduces tail latency but also increases the overall throughput of the systems.

Performance Evaluation of NoSQL Document Databases: Couchbase, CouchDB, and MongoDB

NoSQL document databases emerged as an alternative to relational databases for managing large volumes of data. NoSQL document databases ensure big data storage and good query performance and are essential when the data scheme does not fit into the scheme of relational databases. They store their data in the form of documents and can handle unstructured, semi-structured, and structured data. This work evaluates the top three open-source NoSQL document databases: Couchbase, CouchDB, and MongoDB with Yahoo! Cloud Serving Benchmark (YCSB), which has become a standard for NoSQL database evaluation. The performance and scale-up of document databases are assessed using YCSB workloads with a different number of records and threads, where the runtime is measured for each database. In the experimental evaluation, we concluded that MongoDB is the database with the best runtime, except for the workload composed by scan operations. In addition, we identified CouchDB as the database with the best scale-up when varying the number of threads.

Benchmarking Redis and HBase NoSQL Databases using Yahoo Cloud Service Benchmarking tool

The Not Structured Query Language (NoSQL) databases have become more relevant to applications developers as the need for scalable and flexible data storage for online applications has increased. Each NoSQL database system provides features that fit particular types of applications. Thus, the developer must carefully select according to the application’s needs. Redis is a key-value NoSQL database that provides fast data access. On the other hand, the Apache HBase database is a column-oriented database that offers scalability and fast data access, is a promising alternative to Redis in some types of applications. In this research paper, the goal is to use the Yahoo Cloud Serving Benchmark (YCSB) to compare the performance of two databases (Redis and HBase). The YCSB platform has been developed to determine the throughput of both databases against different workloads. This paper evaluates these NoSQL databases with six workloads and varying threads.

Auto-scaling containerized cloud applications: A workload-driven approach

Today, cloud computing presents new business opportunities as it offers various technological advantages including elastic computing and efficient pricing strategies. Although, cloud users have access to large amount of resources, it is yet a challenging task to efficiently manage the hardware resources in a cloud environment. In this article, we present PACE (Performance-aware Auto-scaler for Cloud Elasticity), a framework for auto-scaling containerized cloud applications based on workload demand. The framework offers a) reactive auto-scaling using threshold-based rules to avoid application failures during intensive workload tasks and b) proactive auto-scaling using convolutional neural networks (CNN) and K-means to generate elastic scaling policies that incorporate future workload demands. The experimental analysis is based on the Yahoo! Cloud Serving Benchmark (YCSB) executed in Redis containers deployed on the Google Cloud Platform. The proposed framework can automatically adjust cloud resources to satisfy workload demand and ensure Quality of Service (QoS) requirements.

PRMS: Design and Development of Patients’ E-Healthcare Records Management System for Privacy Preservation in Third Party Cloud Platforms

In the current digital era, personal data storage on public platforms is a major cause of concern with severe security and privacy ramifications. This is true especially in e-health data management since patient’s health data must be managed following a slew of established standards. The Cloud Service Providers (CSPs) primarily provide computing and storage resources. However, data security in the cloud is still a major concern. In several instances, Blockchain technology rescues the CSPs by providing the robust security to the underlying data by encrypting data using the unique and secret keys. Each network user in Blockchain has its own unique and secret keys linked directly to the transaction keys as a digital signature to protect the data. However, Blockchain technology suffers from the latency and throughput issues in high workload scenarios. To overcome e-healthcare records privacy issues in a third-party cloud, we designed a Patient’s E-Healthcare Records Management System (PRMS) that focuses on latency and throughput. A comprehensive performance analysis of PRMS is carried out on different third-party clouds to validate its applicability. Moreover, the proposed PRMS system is compared with Blockchain platforms such as Hyperledger Fabric v0.6 and Etherium 1.5.8 against latency and throughput by adjusting the workload for each platform up to 10,000 transactions per second. The proposed PRMS is compared to the Secure and Robust Healthcare-Based Blockchain (SRHB) approach using Yahoo Cloud Serving Benchmark (YCSB) and small bank datasets. The experimental results indicate that deploying PRMS on AmazonWeb Services decreases System Execution Time (SET) and the Average Delay (AD) time by 2.4%, 8.33%, and 25.15%, 15.26%, respectively. Additionally, deploying PRMS on the Google Cloud Platform decreases System Execution Time (SET) and Average Delay (AD) by 2.27%, 2.4%, and 2.72%, 4.73% AD, respectively. The experimental results confirm the superiority of the PRMS under the high workload scenario over SRHB and its applicability in cloud data centers.

Comparative Study Between the MySQL Relational Database and the MongoDB NoSQL Database

NoSQL databases are new architectures developed to remedy the various weaknesses that have affected relational databases in highly distributed systems such as cloud computing, social networks, electronic commerce. Several companies loyal to traditional relational SQL databases for several decades seek to switch to the new “NoSQL” databases to meet the new requirements related to the change of scale in data volumetry, the load increases, the diversity of types of data handled, and geographic distribution. This paper develops a comparative study in which the authors will evaluate the performance of two databases very widespread in the field: MySQL as a relational database and MongoDB as a NoSQL database. To accomplish this confrontation, this research uses the Yahoo! Cloud Serving Benchmark (YCSB). This contribution is to provide some answers to choose the appropriate database management system for the type of data used and the type of processing performed on that data.

InK: In-Kernel Key-Value Storage with Persistent Memory

Block-based storage devices exhibit different characteristics from main memory, and applications and systems have been optimized for a long time considering the characteristics in mind. However, emerging non-volatile memory technologies are about to change the situation. Persistent Memory (PM) provides a huge, persistent, and byte-addressable address space to the system, thereby enabling new opportunities for systems software. However, existing applications are usually apt to indirectly utilize PM as a storage device on top of file systems. This makes applications and file systems perform unnecessary operations and amplify I/O traffic, thereby under-utilizing the high performance of PM. In this paper, we make the case for an in-Kernel key-value storage service optimized for PM, called InK. While providing the persistence of data at a high performance, InK considers the characteristics of PM to guarantee the crash consistency. To this end, InK indexes key-value pairs with B+ tree, which is more efficient on PM. We implemented InK based on the Linux kernel and evaluated its performance with Yahoo Cloud Service Benchmark (YCSB) and RocksDB. Evaluation results confirms that InK has advantages over LSM-tree-based key-value store systems in terms of throughput and tail latency.

An efficient resource sharing technique for multi-tenant databases

Multi-tenancy is one of the key components of cloud computing environment. Multi-tenant database system in SaaS (Software as a Service) has gained a lot of attention in academics, research and business arena. These database systems provide scalability and economic benefits for both cloud service providers and customers(organizations/companies referred as tenants) by sharing same resources and infrastructure in isolation of shared databases, network and computing resources with Service level agreement (SLA) compliances. In a multitenant scenario, active tenants compete for resources in order to access the database. If one tenant blocks up the resources, the performance of all the other tenants may be restricted and a fair sharing of the resources may be compromised. The performance of tenants must not be affected by resource-intensive activities and volatile workloads of other tenants. Moreover, the prime goal of providers is to accomplish low cost of operation, satisfying specific schemas/SLAs of each tenant. Consequently, there is a need to design and develop effective and dynamic resource sharing algorithms which can handle above mentioned issues. This work presents a model embracing a query classification and worker sorting technique to efficiently share I/O, CPU and Memory thus enhancing dynamic resource sharing and improvising the utilization of idle instances proficiently. The model is referred as Multi-Tenant Dynamic Resource Scheduling Model (MTDRSM) .The MTDRSM support workload execution of different benchmark such as TPC-C(Transaction Processing Performance Council), YCSB(The Yahoo! Cloud Serving Benchmark)etc. and on different database such as MySQL, Oracle, H2 database etc. Experiments are conducted for different benchmarks with and without SLA compliances to evaluate the performance of MTDRSM in terms of latency and throughput achieved. The experiments show significant performance improvement over existing Mute Bench model in terms of latency and throughput.

EdgeKV: Decentralized, scalable, and consistent storage for the edge

Edge computing moves the computation closer to the data and the data closer to the user to overcome the high latency communication of cloud computing. Storage at the edge allows data access with high speeds that enable latency-sensitive applications in areas such as autonomous driving and smart grid. However, several distributed services are typically designed for the cloud and building an efficient edge-enabled storage system is challenging because of the distributed and heterogeneous nature of the edge and its limited resources. In this paper, we propose EdgeKV, a decentralized storage system designed for the network edge. EdgeKV offers fast and reliable storage, utilizing data replication with strong consistency guarantees. With a location-transparent and interface-based design, EdgeKV can scale with a heterogeneous system of edge nodes. We implement a prototype of the EdgeKV modules in Golang and evaluate it in both the edge and cloud settings on the Grid’5000 testbed. We utilize the Yahoo! Cloud Serving Benchmark (YCSB) to analyze the system’s performance under realistic workloads. Our evaluation results show that EdgeKV outperforms the cloud storage setting with both local and global data access with an average write response time and throughput improvements of 26% and 19% respectively under the same settings. Our evaluations also show that EdgeKV can scale with the number of clients, without sacrificing performance. Finally, we discuss the energy efficiency improvement when utilizing edge resources with EdgeKV instead of a centralized cloud.

Secure and efficient multi-tenant database management system for cloud computing environment

Multi-tenancy is one of the important feature of cloud computing environment. Multi-tenant database system in SaaS (Software as a Service) deployed in cloud infrastructure has gained wide interest in both academics as well as industries, as it provides scalability and economic benefit for both cloud service provider and end users. However, low trust on the rented computational resources prevents users from utilizing the same. Further, in multitenant systems the communication channels and other computational resources are shared, inducing privacy and security issues. As tenants are anonymous in nature, a user may not find a trustworthy co-tenant and tenants depend on cloud service provider to assign trustworthy co-tenants. However, cloud service provider allows maximum co-tenancy irrespective of the behaviors of tenants to maximize resource utilization. Of late, a number of approaches have been presented based on reputation management mechanism to identify good and malicious tenants. However, state-of-art models are not efficient when behavior of malicious tenant changes rapidly. In order to overcome the said research challenge, this work presents a secure and efficient multi-tenant database management system (SEMTDBMS) for cloud computing environment. SEMTDBMS first analyzes security requirement of tenant workers and suggests a secureness weight metric for selection of tenant worker. Subsequently, a novel workload scheduler for scheduling workload among tenants has been presented. Experiments are conducted considering OLTP benchmark such as TPC-C and Yahoo! Cloud Serving Benchmark (YCSB) benchmark with and without security compliances. SEMTDBMS show significant performance improvement in terms of latency and throughput over state-of-art model.

Strict Timed Causal Consistency as a Hybrid Consistency Model in the Cloud Environment

Cloud computing is a model of distributed systems. This system allows users to access virtual resources including the processing power, storage, applications, etc. Storage as a Service (SaaS) is one of the cloud computing services. Cloud storage systems provide this service for the end-users, and deliver data availability and durability as well as global accessibility throughout the Internet. High data availability and scalability are very crucial criteria for the end-users in cloud storage systems. To achieve them, we need replication in these systems. However, the replication brings about asynchronization of data among replicas in different cloud data-centers. It reduces the performance of the cloud storage systems as well. Therefore, replication is one of the crucial challenges in cloud storage systems. These systems need to ensure that the data are synchronized among different replicas by implementing consistency policies. In this paper, we present the Strict Timed Causal Consistency (STCC) as a hybrid consistency model which can be considered as an extension to the cloud computing. This consistency model has two components: client-side, and server-side. At the client-side, this model supports monotonic read, monotonic write, read your write, and write follow read consistencies. At the server-side, it supports the Timed Causal Consistency (TCC) as well. Additionally, it is stronger than the client-centric and is more flexible than the data-centric approaches. In spite of partition tolerance, our proposed method guarantees the consistency and satisfies data availability. Cassandra is a NoSQL database with high scalability and availability. Cassandra comes with multiple consistency levels as a service such as ONE, ALL, QUORUM, etc. We have examined the proposed approach with respect to different consistency levels of Cassandra and Causal Consistency (CC). Yahoo Cloud Serving Benchmark (YCSB) consists of a number of workloads which are used to evaluate our proposed method. We have executed different workloads on the Cassandra clusters and with respect to which we have made a comparison between the performance of our proposed method and the four other different consistency levels in Cassandra. The experimental results based on the comparison between the proposed method and ONE, ALL, QUORUM, as well as the CC consistencies, on a Cassandra cluster with 24 nodes, testify that on average our approach has reduced the stale read rate by 24% on workload-A, and on workload-B by 25%. Also, the system throughput with respect to workload-A has increased by more than 20%. Besides, when we applied our proposed STCC on workload-B the system throughput increased by almost 35%.

Automated Modeling of Real-Time Anomaly Detection using Non-Parametric Statistical technique for Data Streams in Cloud Environments

The main objective of online anomaly detection is to identify abnormal/unusual behavior such as network intrusions, malware infections, over utilized system resources due to design defects etc from real time data stream. Terrabytes of performance data generated in cloud data centers is a well accepted example of such data stream in real time. In this paper, we propose an online anomaly detection framework using non-parametric statistical technique in cloud data center. In order to determine the accuracy of the proposed work, we experiments it to data collected from RUBis cloud testbed and Yahoo Cloud Serving Benchmark (YCSB). Our experimental results shows the greater accuracy in terms of True Positive Rate (TPR), False Positive Rate (FPR), True Negative Rate (TNR) and False Negative Rate (FNR).

Relative Scalability of NoSQL Databases for Genotype Data Manipulation

Genotype data manipulation is one of the greatest challenges in bioinformatics and genomics mainly because of high dimensionality and unbalancing characteristics. These peculiarities explains why Relational Database Management Systems (RDBMSs), the "de facto" standard storage solution, have not been presented as the best tools for this kind of data. However, Big Data has been pushing the development of modern database systems that might be able to overcome RDBMSs deficiencies. In this context, we extended our previous works on the evaluation of relative performance among NoSQLs engines from different families, adapting the schema design in order to achieve better performance based on its conclusions, thus being able to store more SNP markers for each individual. Using Yahoo! Cloud Serving Benchmark (YCSB) benchmark framework, we assessed each database system over hypothetical SNP sequences. Results indicate that although Tarantool has the best overall throughput, MongoDB is less impacted by the increase of SNP markers per individual.

Storage SLA Guarantee with Novel SSD I/O Scheduler in Virtualized Data Centers

Service level agreements (SLAs) for storage performance in virtualized systems are difficult to guarantee, because different consolidated virtual machines have their own performance requirements. Moreover, hard disk drives (HDDs) in virtualized systems are being replaced by solid-state drives (SSDs). SSDs have higher throughput and lower latency than HDDs; however, they pose new challenges in terms of SLAs. In this paper, we determine that existing I/O schedulers working with SSDs fail to guarantee SLAs among virtualmachines, and do not effectively utilize the high performance of SSDs. To address this issue, we propose the opportunistic I/O scheduler (OIOS), a novel I/O scheduler for SSDs. OIOS guarantees SLAs and fully utilizes the high performance of SSDs. To support realistic SLAs, OIOS provides diverse SLA support functions, including reservations, limitations, and proportional sharing. In addition, OIOS accepts SLAs that are specified in four measurement types: bandwidth, I/Os per second (IOPS), latency, and utilization. Experimental results show that OIOS increases the aggregated bandwidth of VMs by 80 percent compared to mClock, while achieving a similar level of fairness. In addition, we evaluate the proposed scheduler with realistic benchmarks, such as Filebench and the Yahoo CloudServing Benchmark. OIOS successfully guarantees the requirements of diverse SLAs with different metrics.

Optimizing data partition for scaling out NoSQL cluster

SummaryData partition impacts the performance of Not Only SQL (NoSQL) systems significantly. Nowadays, many of the peer‐to‐peer NoSQL systems use consistent hashing to partition data automatically. These systems use virtual nodes and random data placement methods to divide the consistent hashing ring, which may lead to imbalanced data partition and degrade the overall system performance. The problem is prominent especially for scaling out heterogeneous clusters. Considering the capacity of each node, an imbalance coefficient of data distribution for a cluster is proposed firstly in this paper. Based on the imbalance coefficient, we propose a dynamic programming algorithm to calculate the position of the new coming node in the consistent hashing ring, which expands the consistent hashing ring more evenly without re‐shuffling the entire datasets. Simulations and experiments on Cassandra with Yahoo! Cloud Serving Benchmark (YCSB) benchmark show our algorithm is better than the state‐of‐the‐art work. Copyright © 2015 John Wiley & Sons, Ltd.

Performance Evaluation of NoSQL Systems using YCSB in a Resource Austere Environment

NoSQL is a database used to store high volume of data. NoSQL databases are horizontally scalable, distributed, open source and non-relational. High performance is a major concern for practically every data-driven system. NoSQL databases claim to deliver faster performance than the popular Relational database systems in various use cases, most notably those involving huge data. While this is always the case, it should be understood that not all NoSQL databases are created alike where performance is concerned. This being the case, IT professionals works hard to ensure that the database they select is optimized for the success of their application use cases. Such selection can be made in-house, based on tests with academic database benchmarks. We present the Yahoo! Cloud Serving Benchmark (YCSB) framework, with the goal of facilitating performance comparisons of the new generation of NoSQL databases in an environment where resources are limited. Unlike many previous benchmarks that considered a cluster or distributed system that NoSQL is known for, we limit out experiment to a single PC assuming a cluster with a single node or a distributed system with a single PC. We define a core set of benchmarks and report results for four widely used systems: MongoDB, ElasticSearch, Redis, and OrientDB implementation.

Graph database benchmarking on cloud environments with XGDBench

Online graph database service providers have started migrating their operations to public clouds due to the increasing demand for low-cost, ubiquitous graph data storage and analysis. However, there is little support available for benchmarking graph database systems in cloud environments. We describe XGDBench which is a graph database benchmarking platform for cloud computing systems. XGDBench has been designed with the aim of creating an extensible platform for graph database benchmarking which makes it suitable for benchmarking future HPC systems. We extend the Yahoo! Cloud Serving Benchmark (YCSB) to the area of graph database benchmarking by creation of XGDBench. The benchmarking platform is written in X10 which is a PGAS language intended for programming future HPC systems. We describe the architecture of the XGDBench and explain how it differs from the current state-of-the-art. We conduct performance evaluation of five famous graph data stores AllegroGraph, Fuseki, Neo4j, OrientDB, and Titan using XGDBench on Tsubame 2.0 HPC cloud environment.