Batch Means Procedure Research Articles

Performance of Skart: A Skewness- and Autoregression-Adjusted Batch Means Procedure for Simulation Analysis

An analysis is given for an extensive experimental performance evaluation of Skart, an automated sequential batch means procedure for constructing an asymptotically valid confidence interval (CI) on the steady-state mean of a simulation output process. Skart is designed to deliver a CI satisfying user-specified requirements on absolute or relative precision as well as coverage probability. Skart exploits separate adjustments to the half-length of the classical batch means CI so as to account for the effects on the distribution of the underlying Student's t-statistic that arise from skewness (nonnormality) and autocorrelation of the batch means. Skart also delivers a point estimator for the steady-state mean that is approximately free of initialization bias. In an experimental performance evaluation involving a wide range of test processes, Skart compared favorably with other steady-state simulation analysis methods—namely, its predecessors ASAP3, WASSP, and SBatch, as well as ABATCH, LBATCH, the Heidelberger–Welch procedure, and the Law–Carson procedure. Specifically, Skart exhibited competitive sampling efficiency and closer conformance to the given CI coverage probabilities than the other procedures, especially in the most difficult test processes.

Performance of Skart

An analysis is given for an extensive experimental performance evaluation of Skart, an automated sequential batch means procedure for constructing an asymptotically valid confidence interval (CI) o...

Skart: A skewness- and autoregression-adjusted batch-means procedure for simulation analysis

Skart is an automated sequential batch-means procedure for constructing a skewness- and autoregression-adjusted confidence interval (CI) for the steady-state mean of a simulation output process either in discrete time (i.e., using observation-based statistics), or in continuous time (i.e., using time-persistent statistics). Skart delivers a CI designed to satisfy user-specified requirements concerning both the CI's coverage probability and its absolute or relative precision. Skart exploits separate adjustments to the classical batch-means CI to account for the effects on the distribution of the underlying Student's t-statistic arising from skewness and autocorrelation of the batch means. The skewness adjustment is based on a Cornish–Fisher expansion for the classical batch-means t-statistic, and the autocorrelation adjustment is based on a first-order autoregressive approximation to the batch-means autocorrelation function. Skart also delivers a point estimator for the steady-state mean that is approximately free of initialization bias. The associated warm-up period is based on iteratively applying Von Neumann's randomness test to spaced batch means with increasing sizes for each batch and its preceding spacer. In extensive experimentation, Skart compared favorably with its competitors. [Supplementary material is available for this article. Go to the publisher's online edition of IIE Transactions for additional discussion, detailed proofs, etc.]

SBatch: A spaced batch means procedure for steady-state simulation analysis

We discuss SBatch, a simplified procedure for steady-state simulation analysis that is based on spaced batch means, incorporating many advantages of its predecessors ASAP3 and WASSP while avoiding many of their disadvantages. SBatch is a sequential procedure designed to produce a confidence-interval (CI) estimator for the steady-state mean response that satisfies user-specified precision and coverage-probability requirements. First SBatch determines a batch size and an interbatch spacer size such that beyond the initial spacer, the spaced batch means approximately form a stationary first-order autoregressive process whose lag-one correlation does not significantly exceed 0.8. Next SBatch delivers a correlation-adjusted CI based on the sample variance and lag-one correlation of the spaced batch means as well as the grand mean of all the individual observations beyond the initial spacer. In an experimental evaluation on a broad range of test problems, SBatch compared favourably with ASAP3 and WASSP.

Performance evaluation of recent procedures for steady-state simulation analysis

The performance of the batch-means procedure ASAP3 and the spectral procedure WASSP is evaluated on test problems with characteristics typical of practical applications of steady-state simulation analysis procedures. ASAP3 and WASSP are sequential procedures designed to produce a confidence-interval estimator for the mean response that satisfies user-specified half-length and coverage-probability requirements. ASAP3 is based on an inverse Cornish-Fisher expansion for the classical batch-means t-ratio, whereas WASSP is based on a wavelet estimator of the batch-means power spectrum. Regarding closeness of the empirical coverage probability and average half-length of the delivered confidence intervals to their respective nominal levels, both procedures compared favorably with the Law-Carson procedure and the original ASAP algorithm. Regarding the average sample sizes required for decreasing levels of maximum confidence-interval half-length, ASAP3 and WASSP exhibited reasonable efficiency in the test problems.

Estimating the steady-state mean from short transient simulations

Steady-state mean performance is a common basis for evaluating discrete event simulation models. However, analysis is complicated by autocorrelation and initial transients. We present confidence interval procedures for estimating the steady-state mean of a stochastic process from observed time series which may be short, autocorrelated, and transient. We extend the generalized least squares estimator of Snell and Schruben [IIE Trans. 17 (1985) 354] and develop confidence interval procedures for single and multiple-replication experiments. The procedures are asymptotically valid and, for short series with reasonable initializations (e.g., empty and idle), are comparable or superior to existing procedures. Further, we demonstrate and explain the robustness of the weighted batch means procedure of Bischak et al. [Manage. Sci. 39 (1993) 1002] to initialization bias. The proposed confidence interval procedure requires neither truncation of initial observations nor choice of batch size, and permits the existence of steady-state mean to be inferred from the data.

ASAP3: a batch means procedure for steady-state simulation analysis

We introduce ASAP3, a refinement of the batch means algorithms ASAP and ASAP2, that delivers point and confidence-interval estimators for the expected response of a steady-state simulation. ASAP3 is a sequential procedure designed to produce a confidence-interval estimator that satisfies user-specified requirements on absolute or relative precision as well as coverage probability. ASAP3 operates as follows: the batch size is progressively increased until the batch means pass the Shapiro-Wilk test for multivariate normality; and then ASAP3 fits a first-order autoregressive (AR(1)) time series model to the batch means. If necessary, the batch size is further increased until the autoregressive parameter in the AR(1) model does not significantly exceed 0.8. Next, ASAP3 computes the terms of an inverse Cornish-Fisher expansion for the classical batch means t -ratio based on the AR(1) parameter estimates; and finally ASAP3 delivers a correlation-adjusted confidence interval based on this expansion. Regarding not only conformance to the precision and coverage-probability requirements but also the mean and variance of the half-length of the delivered confidence interval, ASAP3 compared favorably to other batch means procedures (namely, ABATCH, ASAP, ASAP2, and LBATCH) in an extensive experimental performance evaluation.

An Improved Batch Means Procedure for Simulation Output Analysis

We formulate and evaluate the Automated Simulation Analysis Procedure (ASAP), an algorithm for steady-state simulation output analysis based on the method of nonover-lapping batch means (NOBM). ASAP delivers a confidence interval for an expected response that is centered on the sample mean of a portion of a simulation-generated time series and satisfies a user-specified absolute or relative precision requirement. ASAP operates as follows: The batch size is progressively increased until either (a) the batch means pass the von Neumann test for independence, and then ASAP delivers a classical NOBM confidence interval; or (b) the batch means pass the Shapiro-Wilk test for multivariate normality, and then ASAP delivers a correlation-adjusted confidence interval. The latter adjustment is based on an inverted Cornish-Fisher expansion for the classical NOBM t-ratio, where the terms of the expansion are estimated via an autoregressive-moving average time series model of the batch means. After determining the batch size and confidence-interval type, ASAP sequentially increases the number of batches until the precision requirement is satisfied. An extensive experimental study demonstrates the performance improvements achieved by ASAP versus well-known batch means procedures, especially in confidence-interval coverage probability.