Empirical analysis of the performance of variance estimators in sequential single-run Ranking & Selection: The case of Time Dilation algorithm

Abstract

Ranking and Selection has acquired an important role in the Simulation-Optimization field, where the different alternatives can be evaluated by discrete event simulation (DES). Black box approaches have dominated the literature by interpreting the DES as an oracle providing i.i.d. observations. Another relevant family of algorithms, instead, runs each simulator once and observes time series. This paper focuses on such a method, Time Dilation with Optimal Computing Budget Allocation (TD-OCBA), recently developed by the authors. One critical aspect of TD-OCBA is estimating the response given correlated observations. In this paper, we are specifically concerned with the estimator of the variance of the response which plays a crucial role in simulation budget allocation. We propose an empirical analysis over the performance impact on TD-OCBA of several variance estimators involved in resource allocation. Their performances are discussed in the typical probability of correct selection (PCS) framework.