Stream Processing on Clustered Edge Devices

Abstract

The Internet of Things continuously generates avalanches of raw sensor data to be transferred to the Cloud for processing and storage. Due to network latency and limited bandwidth, this vertical offloading model, however, fails to meet requirements of time-critical data-intensive applications which must act upon generated data with minimum time delays. To address such a limitation, this article proposes a novel distributed architecture enabling stream data processing at the edge of the network, broadening the principle of enabling processing closer to data sources adopted by Fog and Edge Computing. Specifically, this architecture extends the Apache NiFi stream processing middleware with support for run-time clustering of heterogeneous edge devices, such that computational tasks can be horizontally offloaded to peer devices and executed in parallel. As opposed to vertical offloading on the Cloud, the proposed solution does not suffer from increased network latency and is thus able to offer 5-25 times faster response time, as demonstrated by the experiments on a run-time license plate recognition system.