Evaluating Gaussian process metamodels and sequential designs for noisy level set estimation

Xiong Lyu,Mickaël Binois,Michael Ludkovski

doi:10.1007/s11222-021-10014-w

Xiong Lyu, Mickaël Binois + Show 1 more

Open Access

https://doi.org/10.1007/s11222-021-10014-w

Copy DOI

Abstract

We consider the problem of learning the level set for which a noisy black-box function exceeds a given threshold. To efficiently reconstruct the level set, we investigate Gaussian process (GP) metamodels. Our focus is on strongly stochastic simulators, in particular with heavy-tailed simulation noise and low signal-to-noise ratio. To guard against noise misspecification, we assess the performance of three variants: (i) GPs with Student-t observations; (ii) Student-t processes (TPs); and (iii) classification GPs modeling the sign of the response. In conjunction with these metamodels, we analyze several acquisition functions for guiding the sequential experimental designs, extending existing stepwise uncertainty reduction criteria to the stochastic contour-finding context. This also motivates our development of (approximate) updating formulas to efficiently compute such acquisition functions. Our schemes are benchmarked by using a variety of synthetic experiments in 1–6 dimensions. We also consider an application of level set estimation for determining the optimal exercise policy of Bermudan options in finance.

Highlights

1.1 Statement of problemMetamodeling has become widespread for approximating black-box functions that arise in applications ranging from engineering to environmental science and finance (Santner et al 2013)
The main goal of this article is to present a comprehensive assessment of Gaussian process (GP)-based surrogates for stochastic contour-finding
Our analysis focuses on the effect of observation noise on contour-finding algorithms and complements Picheny et al (2013b) and Jalali et al (2017), who benchmarked GP metamodels for Bayesian optimization (BO) where the objective is to evaluate maxx f (x)

Summary

Introduction

1.1 Statement of problemMetamodeling has become widespread for approximating black-box functions that arise in applications ranging from engineering to environmental science and finance (Santner et al 2013). Rather than aiming to capture the precise shape of the function over the entire region, in this article we are interested in estimating the level set where the function exceeds some particular threshold. Level set estimation is common in contexts where we need to quantify the reliability of a system or its performance relative to a benchmark. It arises intrinsically in control frameworks where one wishes to rank the payoff from several available actions (Hu and Ludkovski 2017). We consider a setup where the latent f : D → R is a continuous function over a d-dimensional input space D ⊆ Rd. The level set estimation problem consists in classifying every input x ∈ D = S ∪ C according to

Objectives

Methods

Results

Conclusion