Adaptive pooling-based convolution factorization for deploying CNNs on energy-constrained IoT edge devices

Abstract

Convolutional neural networks (CNNs) involve a tremendous amount of multiply-and-accumulate (MAC) computations, leading to energy issues when deployed on energy-constrained IoT edge devices for computer vision purposes. However, a great number of these computations are ineffectual since they lead to meaningless zero activations, which could be avoided if predicted appropriately. This has been the focus of many studies, whereas it has only been done through zero-activation simplifications. This paper goes beyond those simplifications through a convolution factorization approach to convert the MAC-intensive convolution operations into less complex pooling operations. The main characteristic of the proposed idea is that, given a target accuracy, the applicability and type of factorization are adaptively decided for each individual activation. This is achieved by providing a lightweight multi-class CNN-based predictor, namely RedZAP. The selected types of pooling operations for our convolution factorizations are supported by existing CNN architectures. The experimental results show that, given a top-1 accuracy loss constraint of 1%, the proposed approach saves another 8% MAC operations in comparison to the existing zero-prediction approaches in the literature.