Sequential Probability Ratio Tests: Conservative and Robust

Abstract

In practice, most computers generate simulation outputs sequentially, so it is attractive to analyze these outputs through sequential statistical methods such as sequential probability ratio tests (SPRTs). We investigate several SPRTs for choosing between two hypothesized values for the mean output (response). One SPRT is published in Wald (1945), and allows general distribution types. For a normal (Gaussian) distribution this SPRT assumes a known variance, but in our modified SPRT we estimate the variance. Another SPRT is published in Hall (1962), and assumes a normal distribution with an unknown variance estimated from a pilot sample. We also investigate a modification, replacing this pilot-sample estimator by a fully sequential estimator. We present a sequence of Monte Carlo experiments for quantifying the performance of these SPRTs. In experiment #1 the simulation outputs are normal. This experiment suggests that Wald (1945)’s SPRT with estimated variance gives significantly high error rates. Hall (1962)’s original and modified SPRTs are conservative; i.e., the actual error rates are much smaller than the prespecified (nominal) rates. The most efficient SPRT is our modified Hall (1962) SPRT. In experiment #2 we examine the robustness of the various SPRTs in case of nonnormal output. If we know that the output has a specific nonnormal distribution such as the exponential distribution, then we may also apply Wald (1945)’s original SPRT. Throughout our investigation we pay special attention to the design and analysis of these experiments.