DeepQGHO: Quantized Greedy Hyperparameter Optimization in Deep Neural Networks for on-the-Fly Learning

Abstract

On-the-fly learning is unavoidable for applications that demand instantaneous deep neural network (DNN) training or where transferring data to the central system for training is costly. Hyperparameter optimization plays a significant role in the performance and reliability in deep learning. Many hyperparameter optimization algorithms have been developed for obtaining better validation accuracy in DNNs. Most state-of-the-art hyperparameter optimization techniques are computationally expensive due to the focus on validation accuracy. Therefore, they are unsuitable for on-the-fly learning applications that require faster computation on resource constraint devices (e.g., edge devices). In this paper, we develop a novel greedy approach-based hyperparameter optimization (GHO) algorithm enabling faster computing on edge devices for on-the-fly learning applications. In GHO, we obtain the validation accuracy locally for each of the hyperparameter configurations. The GHO algorithm optimizes each hyperparameter while keeping the others constant in order to converge to the locally optimal solution in the expectation that this choice will lead to a globally optimal solution. We perform an empirical study to compute the performance such as computation time and energy consumption of the GHO and compare it with two state-of-the-art hyperparameter optimization algorithms. We also deploy the GHO algorithm in an edge device to validate the performance of our algorithm. We perform post-training quantization on the GHO algorithm to reduce the inference time. Our GHO algorithm is more than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3\times $ </tex-math></inline-formula> energy efficient and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2\times $ </tex-math></inline-formula> faster than two state-of-the-art hyperparameter optimization techniques on both DNNs and datasets studied in this paper.