A procedure to select the best subset among simulated systems using economic opportunity cost

Abstract

We consider subset selection problems in ranking and selection with tight computational budgets. We develop a new procedure that selects the best m out of k stochastic systems. Previous approaches have focused on individually separating out the top m from all the systems being considered. We reformulate the problem by casting all m-sized subsets of the k systems as the alternatives of the selection problem. This reformulation enables our derivation to follow along traditional ranking and selection frameworks. In particular, we extend the value of information procedure to subset selection. Furthermore, unlike previous subset selection efforts, we use an expected opportunity cost (EOC) loss function as evidence for correct selection. In minimizing the EOC, we consider both deriving an asymptotic allocation rule as well as approximately solving the underlying optimization problem. Experiments show the advantage of our approach for tests with small computational budgets.