A Sketching Approach for Obtaining Real-Time Statistics Over Data Streams in Cloud

Abstract

Many applications of complex event processing (CEP) in Cloud can tolerate analytical errors to some extent, and it provides us an opportunity to optimize real-time analytics using methods of approximate query processing over big data streams. In this article, we present a novel rules-based sampling technique, which supports to construct sketch over one-pass and high-speed asynchronous data streams and provides accurate answers for different types of analytical queries. Moreover, we propose two methods of distributed sketching implementation, i.e., D-AQP <inline-formula><tex-math notation="LaTeX">$_b$</tex-math></inline-formula> and D-AQP <inline-formula><tex-math notation="LaTeX">$_i$</tex-math></inline-formula> , to make our approach to be compatible with batch processing and interactive processing architectures respectively, and be appropriate for stream processing systems in Cloud. Experimental results with real-world and synthetic datasets indicate that our approach can obtain more accurate estimates and improve two times of system throughput when compared with state-of-the-art Hadoop-based approximate engine BlinkDB. When compared with current batch processing systems Spark and stream processing system Spark-Streaming, our methods of D-AQP <inline-formula><tex-math notation="LaTeX">$_b$</tex-math></inline-formula> and D-AQP <inline-formula><tex-math notation="LaTeX">$_i$</tex-math></inline-formula> can achieve 2 and 4 orders of magnitude improvement on query response time respectively.