Abstract
The rapid growth of stream applications in financial markets, health care, education, social media, and sensor networks represents a remarkable milestone for data processing and analytic in recent years, leading to new challenges to handle Big Data in real-time. Traditionally, a single cloud infrastructure often holds the deployment of Stream Processing applications because it has extensive and adaptative virtual computing resources. Hence, data sources send data from distant and different locations of the cloud infrastructure, increasing the application latency. The cloud infrastructure may be geographically distributed and it requires to run a set of frameworks to handle communication. These frameworks often comprise a Message Queue System and a Stream Processing Framework. The frameworks explore Multi-Cloud deploying each service in a different cloud and communication via high latency network links. This creates challenges to meet real-time application requirements because the data streams have different and unpredictable latencies forcing cloud providers’ communication systems to adjust to the environment changes continually. Previous works explore static micro-batch demonstrating its potential to overcome communication issues. This paper introduces BurstFlow, a tool for enhancing communication across data sources located at the edges of the Internet and Big Data Stream Processing applications located in cloud infrastructures. BurstFlow introduces a strategy for adjusting the micro-batch sizes dynamically according to the time required for communication and computation. BurstFlow also presents an adaptive data partition policy for distributing incoming streams across available machines by considering memory and CPU capacities. The experiments use a real-world multi-cloud deployment showing that BurstFlow can reduce the execution time up to 77% when compared to the state-of-the-art solutions, improving CPU efficiency by up to 49%.
Highlights
INTRODUCTIONThe advent of the Internet of Things (IoT) has led to new challenges in the Big Data era due to the limitations of storage, computation, and communication of existing devices since IoT devices generate massive amounts of data that require processing to support the decision-making
The proposed solution overcomes existing orchestration issues presented in cloud-based stream processing frameworks
BurstFlow enables to control the distribution of data in each operator replica by employing ad-hoc partitioning policies
Summary
The advent of the Internet of Things (IoT) has led to new challenges in the Big Data era due to the limitations of storage, computation, and communication of existing devices since IoT devices generate massive amounts of data that require processing to support the decision-making. The SP applications consume one message at-at-time from the MQS because they assume the data source is co-located in the data center, neglecting the existing network latency Another issue is that cloud-based frameworks count on homogeneous workloads when distributing data in the SP framework and neglect performance metrics such as data ingestion, memory, or network utilization. Unlike the works found in the literature, BurstFlow uses an adaptive and dynamic model to estimate the number of events per message based on feedback loops that monitor the batch size in the memory of workers to further forward data. This approach leads to overcoming contention scenarios while maintaining network stability. BurstFlow explores this gap to propose a flow partition approach
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
