Enabling Large Scale Deep Learning on Smart Device by Exploiting Edge-Cloud Computational Paradigm

Abstract

A deep neural network (DNN) has the ability to rival humans in performing wide variety of tasks. The exceptional ability of DNN comes with high computational, memory, and power cost. In order to enable DNN on edge devices and any realtime environments like self-driving car, many techniques have been developed. Most of the techniques use a single network and sacrifice huge amount of accuracy to reduce the cost. In our paper, we propose a hierarchical network to enable computation on edge devices with the help of a cloud to reduce the accuracy loss. The networks have to be able to detect uncertainty in their prediction which is not usually calibrated. This paper uses modern portfolio theory to calibrate uncertainty on the model prediction. The idea is to optimize the return of a horse race problem based on the doubling rate of gambling where a gambler needs to find the proportion between making a prediction and abstain when the network is not confident. For the high abstain score, the data will be forwarded to a deeper model for better prediction. We demonstrated that the proposed framework outperforms knowledge distillation and pruning algorithms. The superiority of our result enables a high-quality result on intelligence IoT applications.