Content-Aware Adaptive Device–Cloud Collaborative Inference for Object Detection

Abstract

Many intelligent applications based on deep neural networks are increasingly running on Internet of Things (IoT) devices. Unfortunately, the computing resources of these IoT devices are limited, which will seriously hinder the widespread deployment of various smart applications. A popular solution is to offload part of computation tasks from IoT device to cloud by way of device-cloud collaboration. However, existing collaboration approaches may suffer from long network transmission delay or degraded accuracy due to the large amount of intermediate results, bring enormous challenges to the tasks such as object detection that require massive computing resources. In this paper, we propose an efficient Device-Cloud Collaborative Inference (DCCI) object detection framework, which dynamically adjusts the amount of transferred data according to the content of input images. Specifically, a content-aware hard-case discriminator is proposed to automatically classify the input images as hard-cases or simple-cases, the hard-cases are uploaded to the cloud to be processed by a deployed heavyweight model, and the simple cases are processed by a light-weight model deployed to the IoT device, where the light-weight model is automatically compressed based on reinforcement learning according to the resource constraints of the IoT device. Furthermore, a collaborative scheduler based on the run-time load and network transmission capability of IoT devices is proposed to optimize the collaborative computation between IoT devices and the cloud. Extensive experimental evaluations show that compared to the Device-only approach, DCCI can reduce the memory footprint and compute resources of IoT devices by more than 90.0% and 30.87%, respectively. Compared to Cloud-centric, DCCI can save 2.0× of network bandwidth. In addition, compared with the state-of-the-art DNN partitioning method, DCCI can save 1.2× of inference latency, and 1.3× of IoT device energy consumption with the same accuracy constraint.