Evaluating NiFi and MQTT based serverless data pipelines in fog computing environments

Abstract

Recent advancements in Internet of Things (IoT) technology have driven massive amounts of raw data. Due to the hardware limitations of the constraint IoT environments, we cannot fully perform all data analytical tasks in the local environment. In contrast, data transferring, storing, and processing in the cloud suffer from cloud cost, latency and security issues. In this context, we require a multi-layer data processing environment to balance all these issues. Data pipeline concepts allow data processing between source and destination on multiple fog devices. On the other hand, IoT applications are event-driven. So, building these apps in a serverless computing manner minimizes operational and billing costs. Moreover, this provides flexibility to deploy tasks anywhere, irrespective of the underlying operating system. In this paper, we investigated two serverless data pipeline approaches designed with Message Queuing Telemetry Transport (MQTT) and Apache NiFi. Our proposed approaches were tested with the image streaming data, where we performed object detection in the images. Our experimental results compared these two approaches regarding pipeline execution time, memory and CPU usage. The results conclude that the Apache NiFi-based serverless data pipeline consumes more CPU than MQTT-based serverless data pipeline, however outperforms it in terms of execution time and memory utilization.