Getting more from the data in a multinomial selection problem

Abstract

We consider the problem of determining which of k simulated systems is most likely to be the best per former based on some objective performance measure. The standard experiment is to generate v in dependent vector observations (replications) across the k- systems. A classical multinomial selection pro cedure, BEM (Bechhofer, Elmaghraby, and Morse), prescribes a minimum number of replications so that the probability of correctly selecting the true best system meets or exceeds a prespecified probability. Assuming that larger is better, BEM selects as best the system having the largest value of the performance measure in more replications than any other. We propose using these same v replications across k systems to form vk pseudoreplications (no longer in dependent) that contain one observation from each system, and again select as best the system having the largest value of the performance measure in more pseudoreplications than any other. We expect that this new procedure, AVC (all vector comparisons), dominates BEM in the sense that AVC will never require more independent replications than DEM to meet a prespecified probability of correct selection. We present analytical and simulation results to show how AVC fares versus BEM for different underly ing distribution families, different numbers of populations and various values of v. We also present results for the closely related problem of estimating the probability that a specific system is the best.