Effective hyperparameter optimization using Nelder-Mead method in deep learning

Yoshihiko Ozaki,Masaki Yano,Masaki Onishi

doi:10.1186/s41074-017-0030-7

Abstract

In deep learning, deep neural network (DNN) hyperparameters can severely affect network performance. Currently, such hyperparameters are frequently optimized by several methods, such as Bayesian optimization and the covariance matrix adaptation evolution strategy. However, it is difficult for non-experts to employ these methods. In this paper, we adapted the simpler coordinate-search and Nelder-Mead methods to optimize hyperparameters. Several hyperparameter optimization methods were compared by configuring DNNs for character recognition and age/gender classification. Numerical results demonstrated that the Nelder-Mead method outperforms the other methods and achieves state-of-the-art accuracy for age/gender classification.

Highlights

The evolution of deep neural networks (DNNs) has dramatically improved the accuracy of character recognition [1], object recognition [2, 3], and other tasks
A hyperparameter optimization problem can be formulated as a stochastic black box optimization problem to minimize a noisy black box objective function f (x): Minimize f (x) (x ∈ χ)
The results demonstrated that Bayesian optimization outperforms manual search by a human expert and random search [7, 8]

Summary

Introduction

The evolution of deep neural networks (DNNs) has dramatically improved the accuracy of character recognition [1], object recognition [2, 3], and other tasks. The their increasing complexity increases the number of hyperparameters, which makes tuning of hyperparameters an intractable task. Search space expands exponentially relative to the number of hyperparameters; such naive methods no longer work well. More sophisticated hyperparameter optimization methods are required. A hyperparameter optimization problem can be formulated as a stochastic black box optimization problem to minimize a noisy black box objective function f (x): Minimize f (x) (x ∈ χ)

Methods

Results

Conclusion