Impact of Embedded Deep Learning Optimizations for Inference in Wireless IoT Use Cases

Abstract

Wireless internet-of-things devices typically transmit data to high-end computing platforms such as edge devices or cloud devices for further processing. However, processing the data at the edge or even on the constrained embedded devices using neural network can eliminate the need for high-throughput links and provides several benefits in terms of latency, reliability, privacy and energy consumption. In this article, we quantify how efficient neural networks can run on embedded devices for three typical wireless use cases. To this end, we give an overview of different optimizations and strategies for embedded deep learning inference on constrained devices. Next, we quantify the performance impact of optimized neural networks for edge and embedded inference, which perform up to 2.5x and 20x faster and consume 20x less energy at the cost of less than 2 percent accuracy difference for classification models. Although most published oversized models cannot run on typical embedded devices, with optimizations, we achieve efficient embedded inference and mitigate the need for raw data transmissions and thus preserve privacy. Finally, we discuss trends found in embedded deep learning use cases and present insights between design- and run-time metrics to predict model memory, storage and energy consumption together with model inference time.