Experimental performance evaluation of batch means procedures for simulation output analysis

Abstract

We summarize the results of an extensive experimental performance evaluation of selected batch means procedures for building a confidence interval for a steady-state expected simulation response. We compare the performance of the well-known ABATCH and LBATCH procedures versus ASAP, a recently proposed variant of the method of nonoverlapping batch means (NOBM) that operates as follows: the batch size is progressively increased until either (a) the batch means pass the von Neumann test for independence, and then ASAP delivers a classical NOBM confidence interval; or (b) the batch means pass the Shapiro-Wilk test for multivariate normality, and then ASAP delivers a correlation-adjusted confidence interval. The latter correction is based on an inverted Cornish-Fisher expansion for the classical NOBM t-ratio, where the terms of the expansion are estimated via an autoregressive-moving average time series model of the batch means. Applying ABATCH, ASAP, and LBATCH to the analysis of a suite of twenty test problems involving discrete-time Markov chains, time-series processes, and queueing systems, we found ASAP to deliver confidence intervals that not only satisfy a user-specified absolute or relative precision requirement but also frequently outperform the corresponding confidence intervals delivered by ABATCH and LBATCH with respect to coverage probability.