Resilient Stream Processing in Edge Computing

Abstract

The proliferation of Internet-of-Things (IoT) devices is rapidly increasing the demands for efficient processing of low latency stream data generated close to the edge of the network. A large number of IoT applications require continuous processing of data streams in real-time. Examples include virtual reality applications, connected autonomous vehicles and smart city applications. Although current distributed stream processing systems offer various forms of fault tolerance, existing schemes do not understand the dynamic characteristics of edge computing infrastructures and the unique requirements of edge computing applications. Optimizing fault tolerance techniques to meet latency requirements while minimizing resource usage becomes a critical dimension of resource allocation and scheduling when dealing with latency-sensitive IoT applications in edge computing. In this paper, we present a novel resilient stream processing framework that achieves system-wide fault tolerance while meeting the latency requirements for edge-based applications. The proposed approach employs a novel resilient physical plan generation for stream queries and optimizes the placement of operators to minimize the processing latency during recovery and reduces the overhead of checkpointing. We implement a prototype of the proposed techniques in Apache Storm and evaluate it in a real testbed. Our results demonstrate that the proposed approach is highly effective and scalable while ensuring low latency and low-cost recovery for edge-based stream processing applications.