Efficient Deep Learning Hyperparameter Tuning Using Cloud Infrastructure: Intelligent Distributed Hyperparameter Tuning with Bayesian Optimization in the Cloud

Abstract

The paper discusses how we can leverage cloud infrastructure for efficient hyperparameter tuning of deep neural networks on high dimensional hyperparameter spaces using Bayesian Optimization. The paper experiments Bayesian optimization in the cloud at different levels of concurrency for the warmup runs for the Bayesian optimization. Two different distributed hyperparameter tuning approaches were experimented in the cloud - Training on multiple nodes with higher warm-up concurrency Vs Distributed Training on multiple nodes with Horovod and reduced number of warm-up runs. The results indicate that greater number of warm-up runs for Bayesian optimization results in better exploration of the search space. The hyper parameter tuning and distributed training with Horovod in the cloud was performed using the HyperDrive framework of Azure Machine Learning Service for Video Activity Recognition problem. The experiment used a Long-term Recurrent Convolutional Network (LRCN) with transfer learning from Resnet50 backbone.