Smart distributed query execution over data streams

Abstract

Current era is witnessing a tremendous growth in the volume of data that is being generated in the form of data streams by the omnipresent sensors, micro-blogs, e-businesses, etc. Many organizations require on-line processing of their data for real time analysis and actionable alerts. It is not possible to process such voluminous and velocious data in real time using the traditional centralized stream processing engines. Hence distributed stream processing has emerged to facilitate such large scale real time processing. In this work we present a smart distributed event-driven stream processing approach. In contrast to the ordinary stream processing, event-driven stream processing generates query results on the occurrence of specified events only. In the basic event-driven stream processing, even when no event is raised input stream tuples are continuously processed by query operators, though they do not generate any query result. This results in increased system load and wastage of system resources. Whereas in the smart event-driven stream processing scheme, incoming tuples are processed in the presence of events only resulting in reduced system load. The proposed smart distributed event-driven stream processing utilizes the concept of smart query execution to distribute the data stream among the distributed worker nodes in the presence of events only; while in the absence of events no data is distributed as it can not generate query output. This smart data distribution can significantly reduce the network traffic in the absence of events and ultimately results in improved overall system throughput. Detailed experiments are performed to prove the effectiveness of the proposed framework.