Parallel predictive entropy search for multi-objective Bayesian optimization with constraints applied to the tuning of machine learning algorithms

Abstract

Real-world problems often involve the optimization of several objectives under multiple constraints. An example is the hyper-parameter tuning problem of machine learning algorithms. For example, minimizing both an estimate of the generalization error of a deep neural network and its prediction time. We may also consider, as a constraint, that the deep neural network must be implemented in a chip with an area below some size. Here, both the objectives and the constraint are black boxes, i.e., functions whose analytical expressions are unknown and are expensive to evaluate. Bayesian optimization (BO) methods have shown state-of-the-art results in these tasks. For this, they evaluate iteratively, at carefully chosen locations, the objectives and the constraints with the goal of solving the optimization problem in a small number of iterations. Nevertheless, most BO methods are sequential and perform evaluations at just one input location, at each iteration. Sometimes, however, we may evaluate several configurations in parallel. If this is the case, as when a cluster of computers is available, sequential evaluations result in a waste of resources. To avoid this, one has to choose which locations to evaluate in parallel, at each iteration. This article introduces PPESMOC, Parallel Predictive Entropy Search for Multi-objective Bayesian Optimization with Constraints, an information-based batch method for the simultaneous optimization of multiple expensive-to-evaluate black-box functions under the presence of several constraints. Iteratively, PPESMOC selects a batch of input locations at which to evaluate the black-boxes in parallel so as to maximally reduce the entropy of the Pareto set of the optimization problem. To our knowledge, this is the first information-based batch method for constrained multi-objective BO. We present empirical evidence in the form of several optimization problems that illustrate the effectiveness of PPESMOC. Moreover, we also show in several experiments the utility of the proposed method to tune the hyper-parameters of machine learning algorithms.