IPC: Resource and network cost-aware distributed stream scheduling on skewed streams

Abstract

The performance of distributed stream processing engines is significantly compromised when processing stream data with skewed distribution. Current stream partitioning schemes are not able to meet the rigorous requirements of distributed stream processing. We show that network cost is an essential factor for partitioning data, and this factor should be considered when designing a stream partitioning scheme. Additionally, we should efficiently utilize resources in the data partitioning process. Current stream partitioning schemes either use a shuffle grouping approach that efficiently manages workload but faces scalability issues in terms of memory or uses hash-based key grouping schemes that suffer from load balancing issues.We argue that network cost and resource utilization are two crucial factors for stream partitioning schemes. We propose and implement a distributed stream partitioning scheme call IPC that minimizes the network cost and efficiently utilizes resources by leveraging two techniques: process near source and process at local. It also utilizes key splitting and local load estimation techniques to achieve load balancing. We implement the IPC on top of Apache Storm. Experiment results using large scale real-time datasets show that IPC achieves an up to 4.2x improvement in throughput and reduces processing latency by 97% compared to state-of-the-art designs.