Accelerating Hyperparameter Optimization of Deep Neural Network via Progressive Multi-Fidelity Evaluation

Abstract

Deep neural networks usually require careful tuning of hyperparameters to show their best performance. However, with the size of state-of-the-art neural networks growing larger, the evaluation cost of the traditional Bayesian optimization has become unacceptable in most cases. Moreover, most practical problems usually require good hyperparameter configurations within a limited time budget. To speed up the hyperparameter optimization, the successive halving technique is used to stop poorly-performed configurations as early as possible. In this paper, we propose a novel hyperparameter optimization method FastHO, which combines the progressive multi-fidelity technique with successive halving under a multi-armed bandit framework. Furthermore, we employ Bayesian optimization to guide the selection of initial configurations and an efficient data subsampling based method to warm start the surrogate model of Bayesian optimization. Extensive empirical evaluation on a broad range of neural networks and datasets shows that FastHO is not only effective to speed up hyperparameter optimization but also can achieve better anytime performance and final performance than the state-of-the-art hyperparameter optimization methods.