Latency and Energy Consumption of Convolutional Neural Network Models from IoT Edge Perspective

Abstract

The increase in computing power and integration of specialized hardware for Artificial Intelligence (AI) acceleration like Tensor Processing Units (TPU) enable complex machine learning at edge devices in the Internet of Things (IoT). However, wireless portable systems are limited in computing power and battery lifetime. To increase the battery lifetime of edge devices and accelerate inference of IoT systems, many developments focus on combining or outsourcing AI algorithms to a cloud via wireless links e.g. wireless LAN IEEE 802.11ac or mobile network 4G/5G. Due to limitations of restricted wireless transmissions in rural areas mainly below 50 MBit/s, resulting longer transfer times can significantly affect inference latency and energy consumption from the perspective of the IoT edge device and deteriorate the response time of the application. In this work, we provide a prototype setup for image processing via Convolutional Neural Networks (CNN) and investigate inference latency and energy consumption of an IoT edge device with a varying wireless link. The complexity of selected pre-trained CNN models is between 300 MFLOPs to 19.6 GFLOPs where FLOPs are Floating Point Operations. The first experiments address the latency and energy consumption by processing CNN models on the IoT device with and without TPU as edge AI accelerator. Following experiments address the latency and energy consumption on the IoT device in cloud processing mode with and without Graphics Processing Unit (GPU) as cloud AI accelerator. The edge device sends input data and receives the results via wireless link from 1 MBit/s to 50 MBit/s. For CNN models with $${\le }564$$ MFLOPs edge processing with AI acceleration performs better than cloud processing regarding latency and energy efficiency. Even for complex CNN models with 7.6 GFLOPs edge processing can be useful at limited wireless link data rates up to 14 MBit/s. Edge processing without AI acceleration is only an option for low complexity ( $${\le }300$$ MFLOPs) and low expected wireless link data rates.