Ranking and Selection

Abstract

We introduce a new sampling scheme for selecting the best alternative out of a given set of systems that are evaluated with respect to their expected performances. We assume that the systems are simulated on a computer and that a joint observation of all systems has a multivariate normal distribution with unknown mean and unknown covariance matrix. In particular, the observations of the systems may be stochastically dependent as is the case if common random numbers are used for simulation. In each iteration of the algorithm, we allocate a fixed budget of simulation runs to the alternatives. We use a Bayesian set-up with a noninformative prior distribution and derive a new closed-form approximation for the posterior distributions that allows provision of a lower bound for the posterior probability of a correct selection (PCS). Iterations are continued until this lower bound is greater than 1−α for a given α. We also introduce a new allocation strategy that allocates the available budget according to posterior error probabilities. Our procedure needs no additional prior parameters and can cope with different types of ranking and selection tasks. Our numerical experiments show that our strategy is superior to other procedures from the literature, namely, KN ++ and P luck . In all of our test scenarios, these procedures needed more observation and/or had an empirical PCS below the required 1−α. Our procedure always had its empirical PCS above 1−α, underlining the practicability of our approximation of the posterior distribution.