Adaptive and Collaborative Edge Inference in Task Stream with Latency Constraint

Abstract

With the rapid development of the Internet of Things (IoT), more and more smart devices are connected to the Internet, implementing Deep Neural Network (DNN) models on edges for collaborative inference via device-edge synergy has become a feasible method for improving application performance in many scenarios. However, when faced with the task stream scenario with latency guarantee such as video surveillance and industrial production line, we need adaptive edge intelligence to make adjustments in real-time according to the changes of the task stream. There are many adaptive edge intelligence technologies in the existing works, such as early-exit mechanism and model selection, but they don’t take the requirements of the task stream scenario into consideration. In this paper, we propose a device-edge collaborative inference system based on the early-exit mechanism to solve the problem of adaptive edge intelligence in the task stream scenario. Then, we design an offline dynamic programming (DP) algorithm and an online deep reinforcement learning (DRL) algorithm to dynamically select the exit point and partition point of the branchy model in the task stream, which aims to balance the number of tasks accomplished and task inference accuracy in the system. Experimental results show that the DRL algorithm can achieve performance close to that of the DP algorithm in the task stream scenario.