High Dimensional Bayesian Optimization Research Articles

Projected Gaussian Markov Improvement Algorithm for High-Dimensional Discrete Optimization via Simulation

This article considers a discrete optimization via simulation (DOvS) problem defined on a graph embedded in the high-dimensional integer grid. Several DOvS algorithms that model the responses at the solutions as a realization of a Gaussian Markov random field (GMRF) have been proposed exploiting its inferential power and computational benefits. However, the computational cost of inference increases exponentially in dimension. We propose the projected Gaussian Markov improvement algorithm (pGMIA), which projects the solution space onto a lower-dimensional space creating the region-layer graph to reduce the cost of inference. Each node on the region-layer graph can be mapped to a set of solutions projected to the node; these solutions form a lower-dimensional solution-layer graph. We define the response at each region-layer node to be the average of the responses within the corresponding solution-layer graph. From this relation, we derive the region-layer GMRF to model the region-layer responses. The pGMIA alternates between the two layers to make a sampling decision at each iteration. It first selects a region-layer node based on the lower-resolution inference provided by the region-layer GMRF, then makes a sampling decision among the solutions within the solution-layer graph of the node based on the higher-resolution inference from the solution-layer GMRF. To solve even higher-dimensional problems (e.g., 100 dimensions), we also propose the pGMIA+: a multi-layer extension of the pGMIA. We show that both pGMIA and pGMIA+ converge to the optimum almost surely asymptotically and empirically demonstrate their competitiveness against state-of-the-art high-dimensional Bayesian optimization algorithms.

PG-LBO: Enhancing High-Dimensional Bayesian Optimization with Pseudo-Label and Gaussian Process Guidance

Variational Autoencoder based Bayesian Optimization (VAE-BO) has demonstrated its excellent performance in addressing high-dimensional structured optimization problems. However, current mainstream methods overlook the potential of utilizing a pool of unlabeled data to construct the latent space, while only concentrating on designing sophisticated models to leverage the labeled data. Despite their effective usage of labeled data, these methods often require extra network structures, additional procedure, resulting in computational inefficiency. To address this issue, we propose a novel method to effectively utilize unlabeled data with the guidance of labeled data. Specifically, we tailor the pseudo-labeling technique from semi-supervised learning to explicitly reveal the relative magnitudes of optimization objective values hidden within the unlabeled data. Based on this technique, we assign appropriate training weights to unlabeled data to enhance the construction of a discriminative latent space. Furthermore, we treat the VAE encoder and the Gaussian Process (GP) in Bayesian optimization as a unified deep kernel learning process, allowing the direct utilization of labeled data, which we term as Gaussian Process guidance. This directly and effectively integrates the goal of improving GP accuracy into the VAE training, thereby guiding the construction of the latent space. The extensive experiments demonstrate that our proposed method outperforms existing VAE-BO algorithms in various optimization scenarios. Our code will be published at https://github.com/TaicaiChen/PG-LBO.

Gradient-Enhanced Kriging for High-Dimensional Bayesian Optimization with Linear Embedding

This paper explores the application of gradient-enhanced (GE) kriging for Bayesian optimization (BO) problems with a high-dimensional parameter space. We utilize the active subspace method to embed the original parameter space in a low-dimensional subspace. The active subspace is detected by analyzing the spectrum of the empirical second-moment matrix of the gradients of the response function. By mapping the training data onto their respective subspace, the objective function and the constraint functions are efficiently approximated with low-dimensional GE-kriging models. In each cycle of the BO procedure, a new point is found by maximizing the constrained expected improvement function within a low-dimensional polytope, and it is mapped back to the original space for model evaluation. In this way, the computational costs are significantly reduced when compared with standard GE-kriging. We illustrate and assess the proposed approach with numerical experiments ranging from academic benchmark problems to aerodynamic engineering applications. The experimental results show that the proposed method is promising for optimizing high-dimensional expensive objective functions, especially for problems that exhibit a clear low-dimensional active subspace.

A paralleled embedding high-dimensional Bayesian optimization with additive Gaussian kernels for solving CNOP

Conditional Nonlinear Optimal Perturbation (CNOP) is widely used in atmospheric and oceanic predictability studies. Solving CNOP is essentially a nonlinear optimization problem with certain constraints. One method to solve CNOP is the intelligent optimization algorithm. But it may not always obtain the satisfactory solution and efficiency because of the local exploitation of individual particles. The Bayesian optimization algorithm can avoid this local stagnation by building a global probability grasp for the optimization space of CNOP. Nonetheless, the limit of approximately 20-dimensional optimization space hampers its application in solving CNOP of high-dimensional numerical models. To overcome this bottleneck, we propose a paralleled embedding high-dimensional Bayesian optimization with additive Gaussian kernels (PEBOA) algorithm, which mainly consists of the feature extraction process and low-dimensional optimization process. In PEBOA, a feature extraction method (deepFE) using the convolutional Autoencoder with residual connections and customized constraint-preserved loss function is designed, which compresses 10 million dimensional data to relatively low-dimensional search spaces, from tens to hundreds of dimensions. Then, we propose the strategy of additive Gaussian kernels, with which the Bayesian optimization algorithm can optimize in relatively low-dimensional search spaces. Concretely, a kernel handles a subspace, and a subspace can handle dimensions approximately not more than 20. Additionally, the modified acquisition function can sample multiple candidates simultaneously with the aim of accelerating the optimization process. We apply PEBOA to solve CNOP of Regional Ocean Modeling System (ROMS) for identifying optimal initial errors of upstream Kuroshio transport variation, which is a 10 million dimensional and time-consuming problem. The computational performance of PEBOA and physical mechanisms of the obtained CNOP are analyzed. Experimental results indicate that deepFE excels Principal Component Analysis (PCA) in terms of relative error ratio, the magnitude of objective function values, and the obtained CNOP pattern. Besides, compared to deepFE-based Particle Swarm Optimization, PEBOA has better solving efficiency with 3.4% larger objective function values and 2.2 times greater likelihood of obtaining the valid CNOP. Furthermore, the modified acquisition function reduces computation time by about 71.0% with four cores. The physical mechanism analysis shows that CNOPs obtained by PEBOA are almost identical to the adjoint method, which can cause an anomalous increase (decrease) in upstream Kuroshio transport and maintain physics consistency.

High-Dimensional Bayesian Optimization for Analog Integrated Circuit Sizing Based on Dropout and gm/ID Methodology

Bayesian optimization (BO) is popular for a analog circuit sizing problem recently. However, BO can only work well in small-scale circuit. Scaling BO to common circuit optimization (typically dimension > 10) is difficult due to the curse of dimensionality. In this article, we proposed a new BO algorithm (cBO), which can scale BO to more larger scale circuit and improve the optimization result. First, according to the fact that a specification of the analog integrated circuit is mainly determined by a few transistors, dropout strategy is adopted in our algorithm. At each iteration, only a subset of selected variables to be optimized. A variables selection strategy based on mutual information analysis and a new fill-in strategy are proposed. Second, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$g_{m}/I_{D}$ </tex-math></inline-formula> methodology-based optimization strategy is proposed. Optimization variables are not width and the length of transistors, but <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$g_{m}/I_{D}$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$I_{D}$ </tex-math></inline-formula> and length. <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$g_{m}$ </tex-math></inline-formula> , <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$I_{D}$ </tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$g_{m}/I_{D}$ </tex-math></inline-formula> have a direct relationship to circuit performance. This is equivalent to search in circuit feature space, which could achieve better performance and handle constraints more easily. Four commonly used circuits, including low dropout regulator, two stage amplifiers, Bandgap voltage reference (VR), and VR are used for validation. Compared with other high-dimensional BO, common BO, and commercial tools (Cadence ADE_GXL), the proposed algorithm is found to produce the best optimization result and best stability. And, the ablation study shows the effectiveness and efficiency of two ingredients of the proposed algorithm.

High-dimensional Bayesian optimization of 23 hyperparameters over 100 iterations for an attention-based network to predict materials property: A case study on CrabNet using Ax platform and SAASBO

Expensive-to-train deep learning models can benefit from an optimization of the hyperparameters that determine the model architecture. We optimize 23 hyperparameters of a materials informatics model, Compositionally-Restricted Attention-Based Network (CrabNet), over 100 adaptive design iterations using two models within the Adaptive Experimentation (Ax) Platform. This includes a recently developed Bayesian optimization (BO) algorithm, sparse axis-aligned subspaces Bayesian optimization (SAASBO), which has shown exciting performance on high-dimensional optimization tasks. Using SAASBO to optimize CrabNet hyperparameters, we demonstrate a new state-of-the-art on the experimental band gap regression task within the materials informatics benchmarking platform, Matbench (∼4.5% decrease in mean absolute error (MAE) relative to incumbent). Characteristics of the adaptive design scheme as well as feature importances are described for each of the Ax models. SAASBO has great potential to both improve existing surrogate models, as shown in this work, and in future work, to efficiently discover new, high-performing materials in high-dimensional materials science search spaces.

Automated Filter Pruning Based on High-Dimensional Bayesian Optimization

Filter pruning is necessary to efficiently deploy convolutional neural networks on edge devices that have limited computational resources and power budgets. With conventional filter pruning techniques, the same pruning rate is manually specified for different convolutional layers, which is suboptimal and time-consuming. To extract the features from the coarse level to the fine level, the number of filters in each layer has various distributions. Therefore, it is unsuitable to utilize the same pruning rate for different functional layers. To address this issue, we propose a high-dimensional Bayesian optimization-based filter pruning (HDBOFP) algorithm, which aims to automatically determine the most appropriate pruning rate for each convolutional layer. In addition, the proposed method can automatically identify optimal pruning-rate combinations without a time-consuming retraining phase. Compared with conventional filter pruning methods, this automated filter pruning technique exhibits a higher efficiency, which improves accuracy and reduces the required human labor. The effectiveness of our automated filter pruning algorithm is validated through two major computer vision applications, namely image classification and object detection. Specifically, when used in combination with ResNet-110 to classify the CIFAR-10 dataset, HDBOFP reduces the required number of float point operations (FLOPs) by more than 62% without affecting accuracy. Similarly, when HDBOFP is added to the YOLOv5l framework to run detection experiments on the MS-COCO 2017 dataset, FLOPs decrease by more than 43% with only a 1.2% loss in mean average precision, which has advanced the previous studies.

High-Dimensional Bayesian Optimization via Tree-Structured Additive Models

Bayesian Optimization (BO) has shown significant success in tackling expensive low-dimensional black-box optimization problems. Many optimization problems of interest are high-dimensional, and scaling BO to such settings remains an important challenge. In this paper, we consider generalized additive models in which low-dimensional functions with overlapping subsets of variables are composed to model a high-dimensional target function. Our goal is to lower the computational resources required and facilitate faster model learning by reducing the model complexity while retaining the sample-efficiency of existing methods. Specifically, we constrain the underlying dependency graphs to tree structures in order to facilitate both the structure learning and optimization of the acquisition function. For the former, we propose a hybrid graph learning algorithm based on Gibbs sampling and mutation. In addition, we propose a novel zooming-based algorithm that permits generalized additive models to be employed more efficiently in the case of continuous domains. We demonstrate and discuss the efficacy of our approach via a range of experiments on synthetic functions and real-world datasets.

Active and Compact Entropy Search for High-Dimensional Bayesian Optimization

Entropy search and its derivative methods are one class of Bayesian Optimization methods that achieve active exploration of black-box functions. They maximize the information gain about the position in the input space where the black-box function gets the global optimum. However, existing entropy search methods suffer from harassment caused by high dimensional optimization problems. On the one hand, the computation for estimating entropies increases exponentially as dimensions increase, which limits the applicability of entropy search to high dimensional problems. On the other hand, many high-dimensional problems have the property that a large number of dimensions have little influence on the objective function, but currently there is no compress mechanism to exclude these redundant dimensions. In this work, we propose Active Compact Entropy Search (AcCES) to fix these defects. Under the guidance of historical evaluation, AcCES actively explores the prevalent inter-dimensional correlations by maximizing the linear or non-linear relationships that may exist between dimensions in the acquisition function, which is ignored by existing Bayesian Optimization methods. In order to build a more compact input space, redundant dimensions are compressed by exploiting inter-dimensional correlations. Experiments demonstrate that AcCES achieves higher query efficiency and optimal results than existing entropy search methods.

High-dimensional Bayesian optimization using low-dimensional feature spaces

Bayesian optimization (BO) is a powerful approach for seeking the global optimum of expensive black-box functions and has proven successful for fine tuning hyper-parameters of machine learning models. However, BO is practically limited to optimizing 10–20 parameters. To scale BO to high dimensions, we usually make structural assumptions on the decomposition of the objective and/or exploit the intrinsic lower dimensionality of the problem, e.g. by using linear projections. We could achieve a higher compression rate with nonlinear projections, but learning these nonlinear embeddings typically requires much data. This contradicts the BO objective of a relatively small evaluation budget. To address this challenge, we propose to learn a low-dimensional feature space jointly with (a) the response surface and (b) a reconstruction mapping. Our approach allows for optimization of BO’s acquisition function in the lower-dimensional subspace, which significantly simplifies the optimization problem. We reconstruct the original parameter space from the lower-dimensional subspace for evaluating the black-box function. For meaningful exploration, we solve a constrained optimization problem.

Trading Convergence Rate with Computational Budget in High Dimensional Bayesian Optimization

Scaling Bayesian optimisation (BO) to high-dimensional search spaces is a active and open research problems particularly when no assumptions are made on function structure. The main reason is that at each iteration, BO requires to find global maximisation of acquisition function, which itself is a non-convex optimization problem in the original search space. With growing dimensions, the computational budget for this maximisation gets increasingly short leading to inaccurate solution of the maximisation. This inaccuracy adversely affects both the convergence and the efficiency of BO. We propose a novel approach where the acquisition function only requires maximisation on a discrete set of low dimensional subspaces embedded in the original high-dimensional search space. Our method is free of any low dimensional structure assumption on the function unlike many recent high-dimensional BO methods. Optimising acquisition function in low dimensional subspaces allows our method to obtain accurate solutions within limited computational budget. We show that in spite of this convenience, our algorithm remains convergent. In particular, cumulative regret of our algorithm only grows sub-linearly with the number of iterations. More importantly, as evident from our regret bounds, our algorithm provides a way to trade the convergence rate with the number of subspaces used in the optimisation. Finally, when the number of subspaces is "sufficiently large", our algorithm's cumulative regret is at most O*(√TγT) as opposed to O*(√DTγT) for the GP-UCB of Srinivas et al. (2012), reducing a crucial factor √D where D being the dimensional number of input space. We perform empirical experiments to evaluate our method extensively, showing that its sample efficiency is better than the existing methods for many optimisation problems involving dimensions up to 5000.

Embedding high-dimensional Bayesian optimization via generative modeling: Parameter personalization of cardiac electrophysiological models.

The estimation of patient-specific tissue properties in the form of model parameters is important for personalized physiological models. Because tissue properties are spatially varying across the underlying geometrical model, it presents a significant challenge of high-dimensional (HD) optimization at the presence of limited measurement data. A common solution to reduce the dimension of the parameter space is to explicitly partition the geometrical mesh. In this paper, we present a novel concept that uses a generative variational auto-encoder (VAE) to embed HD Bayesian optimization into a low-dimensional (LD) latent space that represents the generative code of HD parameters. We further utilize VAE-encoded knowledge about the generative code to guide the exploration of the search space. The presented method is applied to estimating tissue excitability in a cardiac electrophysiological model in a range of synthetic and real-data experiments, through which we demonstrate its improved accuracy and substantially reduced computational cost in comparison to existing methods that rely on geometry-based reduction of the HD parameter space.

High-dimensional Bayesian optimization with projections using quantile Gaussian processes

Key challenges of Bayesian optimization in high dimensions are both learning the response surface and optimizing an acquisition function. The acquisition function selects a new point to evaluate the black-box function. Both challenges can be addressed by making simplifying assumptions, such as additivity or intrinsic lower dimensionality of the expensive objective. In this article, we exploit the effective lower dimensionality with axis-aligned projections and optimize on a partitioning of the input space. Axis-aligned projections introduce a multiplicity of outputs for a single input that we refer to as inconsistency. We model inconsistencies with a Gaussian process (GP) derived from quantile regression. We show that the quantile GP and the partitioning of the input space increases data-efficiency. In particular, by modeling only a quantile function, we overcome issues of GP hyper-parameter learning in the presence of inconsistencies.

Decentralized High-Dimensional Bayesian Optimization With Factor Graphs

This paper presents a novel decentralized high-dimensional Bayesian optimization (DEC-HBO) algorithm that, in contrast to existing HBO algorithms, can exploit the interdependent effects of various input components on the output of the unknown objective function f for boosting the BO performance and still preserve scalability in the number of input dimensions without requiring prior knowledge or the existence of a low (effective) dimension of the input space. To realize this, we propose a sparse yet rich factor graph representation of f to be exploited for designing an acquisition function that can be similarly represented by a sparse factor graph and hence be efficiently optimized in a decentralized manner using distributed message passing. Despite richly characterizing the interdependent effects of the input components on the output of f with a factor graph, DEC-HBO can still guarantee no-regret performance asymptotically. Empirical evaluation on synthetic and real-world experiments (e.g., sparse Gaussian process model with 1811 hyperparameters) shows that DEC-HBO outperforms the state-of-the-art HBO algorithms.