LDSQP: Scalable and Efficient Log Data Storage and Query Processing Scheme for Cloud Data Centers

Abstract

Cloud data centers are one of the most important information infrastructures for most real-world applications involving information technology, and serves millions of user requests daily. To provide various services with Quality-of-Service (QoS) guarantees to users, such as low network latency and fast response time, it is vital that cloud data centers should (1) manage its resources efficiently, and (2) avoid resource overprovisioning in situation such as virtual machine allocation. To achieve these goals, cloud data centers usually employ modern techniques from artificial intelligence or machine learning community to analyze status information of cloud data center, so as to find actionable strategies. In this paper, we propose LDSQP, an efficient scheme for storing and querying status information in cloud data centers, i.e., log data that contains enriched information collected from servers, virtual machines, and containers in cloud data centers. We design scalable cluster structure and row key format for log data record in data storage, and we also propose a compact and efficient query expression to capture user's data requirement. We introduce an efficient query processing algorithm that can minimize the number of table partitions to scan. Experiments on a real cluster show that our proposed LDSQP is efficient in log data management.