A Comparative Study on Bayesian Optimization

Abstract

Abstract Well-known as an effective algorithm for optimizing expensive black-box functions, the popularity of Bayesian Optimization has surged in recent years alongside with the rise of machine learning thanks to its role as the most important algorithm for hyperparameter optimization. Many have used it, few would comprehend, since behind this powerful technique is a plethora of complex mathematical concepts most computer scientists and machine learning practitioners could barely familiarize themselves with. Even its simplest and most traditional building block - Gaussian Process - alone would involve enough advanced multivariate probability that can fill hundreds of pages. This work reviews this powerful algorithm and its traditional components such as Gaussian Process and Upper Confidence Bound in an alternative way by presenting a fresh intuition and filtering the complications of mathematics. Our paper will serve well as a functional reference for applied computer scientists who seek for a quick understanding of the subject to apply the tool more effectively.KeywordsBayesian OptimizationGaussian ProcessUpper Confidence BoundHyperparameter optimizationExpensive black-box functions