Bayesian Optimization over Mixed Type Inputs with Encoding Methods

Abstract

Traditional Bayesian Optimization (BO) algorithms assume that the objective function is defined over numeric input space. To generalize BO for mixed numeric and categorical inputs, existing approaches mainly model or optimize them separately and thus cannot fully capture the relationship among different types of inputs. The complexity incurred by additional operations for the categorical inputs in these approaches can also reduce the efficiency of BO, especially when facing high-cardinality inputs. In this paper, we revisit the encoding approaches, which transfer categorical inputs to numerical ones to form a concise and easy-to-use BO framework. Specifically, we propose the target mean encoding BO (TmBO) and aggregate encoding BO (AggBO), where TmBO transfers each value of a categorical input based on the outputs corresponding to this value, and AggBO encodes multiple choices of a categorical input through several distinct ranks. Different from the prominent one-hot encoding, both approaches transfer each categorical input into exactly one numerical input and thus avoid severely increasing the dimension of the input space. We demonstrate that TmBO and AggBO are more efficient than existing approaches on several synthetic and real-world optimization tasks.