Abstract
Optimizing a black-box, expensive, and multi-extremal function, given multiple approximations, is a challenging task known as multi-information source optimization (MISO), where each source has a different cost and the level of approximation (aka fidelity) of each source can change over the search space. While most of the current approaches fuse the Gaussian processes (GPs) modelling each source, we propose to use GP sparsification to select only “reliable” function evaluations performed over all the sources. These selected evaluations are used to create an augmented Gaussian process (AGP), whose name is implied by the fact that the evaluations on the most expensive source are augmented with the reliable evaluations over less expensive sources. A new acquisition function, based on confidence bound, is also proposed, including both cost of the next source to query and the location-dependent approximation of that source. This approximation is estimated through a model discrepancy measure and the prediction uncertainty of the GPs. MISO-AGP and the MISO-fused GP counterpart are compared on two test problems and hyperparameter optimization of a machine learning classifier on a large dataset.
Highlights
This paper focuses on the situation arising when a black-box, multi-extremal, and expensive function can be optimized by querying multiple information sources which provide less expensive approximations of the original function
When a pairwise Wilcoxon’s test is used, distances from x∗ are not significantly different, but in the case of multi-information source optimization (MISO)-fused Gaussian processes (GPs) with 3 sources compared to Bayesian optimization (BO) performed on f1(x), MISO-augmented Gaussian process (AGP) with 2 sources (p-value
It is important to remark that MISO-fused GP is not the MISO algorithm proposed in Ghoreishi and Allaire (2019): the undesired poor convergence to the actual optimizer might be mitigated by the adoption of their two-step acquisition function, which is anyway more computationally expensive than the GP confidence bound based acquisition function we have proposed
Summary
This paper focuses on the situation arising when a black-box, multi-extremal, and expensive function can be optimized by querying multiple information sources which provide less expensive approximations of the original function. The final goal is to optimize the original function while keeping low the overall cumulated query cost This setting is known as multi-information source optimization (MISO). Poloczek et al (2017) introduced a general notion of model discrepancy to quantify the difference between each source and the function to optimize, depending on the location. GP approximation— GP sparsification—is considered in this study not for reducing computational costs of GP fitting in MISO, but as the core of our algorithm in order to reduce the discrepancy between f (x) and the single model enabling a uniform Bayesian treatment of all the information sources. GP sparsification methods are described in Section 2.2; here, it is important to anticipate that they are aimed at restricting the GP model to a small set of inducing locations which should be large enough to cover the search space in order to avoid variance starvation (Wang et al 2018), and be as small as possible for efficiency
Sign-in/Register to view highlights & summary Sign-in/Register to access full text options 
Talk to us
Join us for a 30 min session where you can share your feedback and ask us any queries you have
Schedule a callDisclaimer: All third-party content on this website/platform is and will remain the property of their respective owners and is provided on "as is" basis without any warranties, express or implied. Use of third-party content does not indicate any affiliation, sponsorship with or endorsement by them. Any references to third-party content is to identify the corresponding services and shall be considered fair use under The CopyrightLaw.
