Indifference-zone Formulation Research Articles

A Non-Parametric Sequential Procedure for the Generalized Partition Problem

In selection and ranking, the partitioning of treatments by comparing them to a control treatment is an important statistical problem. For over eighty years, this problem has been investigated by a number of researchers via various statistical designs to specify the partitioning criteria and optimal strategies for data collection. Many researchers have proposed designs in order to generalize formulations known at that time. One such generalization adopted the indifference-zone formulation to designate the region between the boundaries for “good” and “bad” treatments as the indifference zone. Since then, this formulation has been adopted by a number of researchers to study various aspects of the partition problem. In this paper, a non-parametric purely sequential procedure is formulated for the partition problem. The “first-order” asymptotic properties of the proposed non-parametric procedure are derived. The performance of the proposed non-parametric procedure for small and moderate sample sizes is studied via Monte Carlo simulations. An example is provided to illustrate the proposed procedure.

Ranking and Selection with Covariates for Personalized Decision Making

We consider a problem of ranking and selection via simulation in the context of personalized decision making, in which the best alternative is not universal, but varies as a function of some observable covariates. The goal of ranking and selection with covariates (R&S-C) is to use simulation samples to obtain a selection policy that specifies the best alternative with a certain statistical guarantee for subsequent individuals upon observing their covariates. A linear model is proposed to capture the relationship between the mean performance of an alternative and the covariates. Under the indifference-zone formulation, we develop two-stage procedures for both homoscedastic and heteroscedastic simulation errors, respectively, and prove their statistical validity in terms of average probability of correct selection. We also generalize the well-known slippage configuration and prove that the generalized slippage configuration is the least favorable configuration for our procedures. Extensive numerical experiments are conducted to investigate the performance of the proposed procedures, the experimental design issue, and the robustness to the linearity assumption. Finally, we demonstrate the usefulness of R&S-C via a case study of selecting the best treatment regimen in the prevention of esophageal cancer. We find that by leveraging disease-related personal information, R&S-C can substantially improve patients’ expected quality-adjusted life years by providing a patient-specific treatment regimen.

Fixed-Confidence, Fixed-Tolerance Guarantees for Ranking-and-Selection Procedures

Ever since the conception of the statistical ranking-and-selection (R8S) problem, a predominant approach has been the indifference-zone (IZ) formulation. Under the IZ formulation, R8S procedures are designed to provide a guarantee on the probability of correct selection (PCS) whenever the performance of the best system exceeds that of the second-best system by a specified amount. We discuss the shortcomings of this guarantee and argue that providing a guarantee on the probability of good selection (PGS)—selecting a system whose performance is within a specified tolerance of the best—is a more justifiable goal. Unfortunately, this form of fixed-confidence, fixed-tolerance guarantee has received far less attention within the simulation community. We present an overview of the PGS guarantee with the aim of reorienting the simulation community toward this goal. We examine numerous techniques for proving the PGS guarantee, including sufficient conditions under which selection and subset-selection procedures that deliver the IZ-inspired PCS guarantee also deliver the PGS guarantee.

A curtailed selection procedure for comparing Bernoulli outcomes with a control

ABSTRACT A sequential selection procedure is proposed for comparing experimental Bernoulli populations with a controlled Bernoulli population. The comparison is made according to the success probability. Based on the numbers of successes of the populations, the indifference zone formulation and the subset selection formulation are integrated to select either the best population or a random-sized subset that contains the best population. Observations are taken one at a time and the populations that are no longer comparable are eliminated until either the best is identified or the total number of observations in at least one population reaches a specific upper bound. We show that the proposed sequential procedure satisfies the same probability requirements as does the corresponding fixed-sample-size procedure. Furthermore, the expected sample size from each population for the proposed procedure is significantly smaller than the sample size required by the fixed-sample-size procedure.

New procedures for selecting the k-best exponential populations

ABSTRACTSuppose exponential populations πi with parameters (μi,σi) (i = 1, 2, …, K) are given. The σi can be unknown and unequal. This article discusses how to select the k (≥1) best populations. Under the subset selection formulation, a one-stage procedure is proposed. Under the indifference zone formulation, a two-stage procedure is proposed. An appealing feature of these procedures is that no statistical tables are needed for their implementation.

Chance Constrained Selection of the Best

Selecting the solution with the largest or smallest mean of a primary performance measure from a finite set of solutions while requiring secondary performance measures to satisfy certain constraints is called constrained selection of the best (CSB) in the simulation ranking and selection literature. In this paper, we consider CSB problems with secondary performance measures that must satisfy probabilistic constraints, and we call such problems chance constrained selection of the best (CCSB). We design procedures that first check the feasibility of all solutions and then select the best among all the sample feasible solutions. We prove the statistical validity of these procedures for variations of the CCSB problem under the indifference-zone formulation. Numerical results show that the proposed procedures can efficiently handle CCSB problems with up to 100 solutions, each with five chance constraints.

Optimal Computing Budget Allocation for Complete Ranking

Previous research in ranking and selection focused on selecting the best design and subset selection. Little research has been done for ranking all designs completely. Complete ranking has been applied to design of experiment, random number generator and population-based search algorithms. In this paper, we consider the problem of ranking all designs. Our objective is to develop an efficient simulation allocation procedure that maximizes the probability of correct ranking with fixed limited computing budget. A previous allocation strategy of complete ranking based on indifference zone formulation is conservative and not efficient enough. We use the optimal computing budget allocation framework to further enhance the efficiency and reduce the amount of budget needed to achieve the same probability of correct ranking. Compared with the previous allocation strategy, our proposed allocation rule performs best under different scenarios.

A Framework for Selecting a Selection Procedure

For many discrete simulation optimization applications, it is often difficult to decide which Ranking and Selection (R&S) procedure to use. To efficiently compare R&S procedures, we present a three-layer performance evaluation process. We show that the two most popular performance formulations, namely the Bayesian formulation and the indifference zone formulation, have a common representation analogous to convex risk measures used in mathematical finance. We then specify how a decision maker can impose a performance requirement on R&S procedures that is more adequate for her risk attitude than the indifference zone or the Bayesian performance requirements. Such a performance requirement partitions the space of R&S procedures into acceptable and nonacceptable procedures. The minimal computational budget required for a procedure to become acceptable introduces an easy-to-interpret preference order on the set of R&S policies. We demonstrate with a numerical example how the introduced framework can be used to guide the choice of selection procedure in practice.

Two-Stage Selection of the Best Signal-To-Noise Ratio with Related Approximations

We consider the problem of calibrating k (≥ 2) channels or signal processors and select the best one, that is the one having the largest signal-to-noise ratio (SNR). Assuming a Gaussian distribution for the back-ground white noise or clutter, the problem reduces to selecting from k (≥ 2) normal populations the one with the largest value of the absolute mean. Assuming unknown means and a common unknown variance, we proceed under Bechhofer's 1954 indifference-zone formulation. Since the common variance is unknown, a single-stage procedure guaranteeing a preassigned level of the minimum probability of correct selection (PCS) would not exist. We propose a two-stage selection methodology whose minimum PCS meets the minimum PCS requirement up to the second-order term with respect to the difference of the absolute means. The roles of the proposed approximation are examined both via exact computations and simulations. A realistic example on passive acoustic sonar detection of marine mammals (Abraham 2004) is included with data analyses.

Statistical Identification in Multinomial Models with Sequential Sampling

SYNOPTIC ABSTRACTWe propose an inverse-type sequential method of statistical identification in multinomial models having unequal cell probabilities. Using the indifference-zone formulation and based on the likelihood ratio of decision vectors, a stopping rule is devised that controls the probability of a correct identification, P {CI} and satisfies a preassigned probability level condition P*. By performing a Monte Carlo experiment, the expected sample sizes are obtained and the numerical results of the proposed procedure are presented for illustration.

A SEQUENTIAL PROCEDURE WITH ELIMINATION FOR PARTITIONING A SET OF NORMAL POPULATIONS HAVING A COMMON UNKNOWN VARIANCE

We consider the problem of partitioning a set of normal populations, with respect to a control population, into two subsets according to their unknown means, under the indifference zone formulation. For this problem Tong (1969) constructed a two-stage and purely sequential procedures and recently Datta and Mukhopadhyay (1998) have considered various multistage methodologies emphasizing the second-order asymptotics. However, all these methodologies adopt a vector-at-a-time sampling approach and lack the desirable feature of eliminating treatments from further sampling at an earlier stage. We propose an elimination type sequential procedure using the Paulson's (1964) elimination idea and design a sampling methodology which eliminates “superior” and “inferior” treatments during the sampling, thereby reducing the average sample size considerably. Theoretical results are obtained to show that the proposed elimination-type procedure maintains the probability of correct partition above a pre specified level. The proposed procedure is studied and compared via the Monte Carlo simulation studies with other competitive procedures known in the literature, under both the LFC and a few non-LFC configurations. The elimination-type procedure is found to be vastly more efficient than its competitors in terms of average sample size.

An empirical evaluation of several methods to select the best system

Simulation is an important tool for comparing the performance of several alternative systems. There is therefore significant interest in procedures that efficiently select the best system, where best is defined by the maximum or minimum expected simulation output. In this paper, we examine both two-stage and sequential procedures that represent three structurally different modeling methodologies for allocating simulation replications to identify the best system, and we evaluate them empirically with respect to several measures of effectiveness. Empirical evidence suggests that sequential procedures perform better than their two-stage counterparts, including a heuristic sequential variation on Rinott's procedure. Further, there appears to be significant benefit to using procedures based on a Bayesian, average-case analysis as opposed to the statistically-conservative indifference-zone formulation.

A two-stage procedure for selecting the largest multinomial cell probability when nuisance cell is present

This paper considers the problem of selecting the cell with the largest cell probability of multinomial distribution when one of the cells is considered as nuisance cell. A multinomial distribution is supposed to have three cells for the first attempt to this problem. A Stein-type two-stage procedure is proposed to select the best cell under the two types of indifference-zone formulation in ranking and selection theory. We first identify the configuration at which the probability of correct selection is minimum under the fixed-sample size procedure when the nuisance cell probability is fixed. Then we propose a two-stage procedure to (1) estimate the nuisance cell probability in the first stage, and (2) select the best cell in the second stage. It is shown that the proposed two-stage procedure is asymptotically efficient. The tables of the procedure parameters are provided and a Monte Carlo simulation is given to illustrate the efficiency of the procedure.

Different definitions of Δ-correct selection for the indifference zone formulation

The goal of the indifference zone formulation of selection (Bechhofer, 1954) consists of selecting the t best variants out of k variants with a probability of at least 1 − β if the parameter difference between the t ‘good’ variants and the k − t ‘bad’ variants is not less than Δ. A review of generalized selection goals not using this difference condition is presented. Within some general classes of distributions, the suitable experimental designs for all these selection goals are identical. Similar results are described for the problem of selecting the best variant in comparison with a control, or standard.

Software for selection procedures

PC programs (packages) for selection procedures are described. Most of them provide the sample size determination for the indifference zone formulation in dependence on given precision requirements ( d and β), or the calculation of the distance d between ‘good’ and ‘bad’ populations in dependence on n and β. Some provide the calculation of d in dependence on β or vice versa for the subset formulation. An ‘ideal’ software does not seem to exist yet.

Selecting the best and the second best multinomial cells

Selection problems in a multinomial distribution with inverse sampling are considered. Two goals are studied, goal I is selecting the most probable cell and the second most probable cells, and goal II is selecting the least probable cell and the second least probable cell. Indifference zone formulation is used and the measure of distance is the ratio of corresponding cell probabilities. Type-2 Dirichlet integrals are used (i) to express the probability of a correct selection as an integral with parameters only in the limits of integration, and (ii) to obtain the least favorable configuration.

On selecting the best component of a multivariate normal population

The problem of selecting the component having the largest population mean in a k-variate normal population is discussed when all the means, variances and correlations are unknown. The indifference-zone formulation has been adopted, and we propose a two-stage procedure (PS) that guarantees the nominal value (P∗) of probability of correct selection under the assumption that all the correlations are non-negative. The procedure PS is the only method thus far proposed that will accomplish the desired objective. It is also of some interest that PS is asymptotically (as δ∗ ↛ 0) better than the procedure PR of Dudewicz and Dalal (1975) for k = 2, P∗≤.95, while for k≥3 the procedure PR is better than PS when ρij = 0, i≠j. This comparison is naturally valid when both PR and PS are applicable, and of course there are situations where only PS is applicable.

Conditions under which F. {N } = for tong's adaptive solution to ranking and selection problems

Tong ‘1978’ proposed an adaptive approach as an alternative to the classical indifference-zone formulation of the problems of ranking and selection. With a fixed pre-selected y*‘1/k < y* < 1’ his procedure calls for the termination of vector-at-a-time sampling when the estimated probability of a correct selection exceeds Y* for the first time. The purpose of this note is to show that for the case of two normal populations with common known variance, the expected number of vector-observations required by Tong's procedure to terminate sampling approaches infinity as the two population means approach equality for Y* ≥ 0.8413.It is conjectured that this phenomenon also persists if the two largest of K ≥3 population means approach equality. Since in the typical ranking and selection setting it usually is assumed that the experimenter has no knowledge concerning the differences between the population means, the experimenter who uses Tong's procedure clearly does so at his own risk.

Analyzing the Distractors of Multiple-Choice Test Items or Partitioning Multinomial Cell Probabilities with Respect to a Standard

When analyzing the distractors of multiple-choice test items, it is sometimes desired to determine which of the distractors has a small probability of being chosen by a typical examinee. At present, this problem is handled in an informal manner. In particular, using an arbitrary number of examinees, the probabilities associated with the distractors are estimated and then sorted according to whether the estimated values are above or below a known constant p0 In this paper a more formal framework for solving this problem is described. The first portion of the paper considers the problem from the point of view of designing an experiment. The solution is based on a procedure similar to an indifference zone formulation of a ranking and election problem. A later section considers methods that might be employed in a retrospective study. Brief consideration is also given to how an analysis might proceed when a test item has been altered in some way.

Designing two factor experiments for selecting interactions

This paper considers the problem of devising a single stage procedure for selecting the treatment combination associated with the largest interaction in a two-factor r fx× c experiment having independent normally distributed observations with common known variance. The intuitive procedure based on the best linear unbiased estimators of the population interactions is employed. Initially an indifference zone formulation is used; the problem of determining the least favorable configuration is reduced to a nonlinear programming problem with log concave objective function and a convex polytope as feasible region. A solution technique is introduced in the context of an illustrative example. The problem is also considered using a preferred population formulation; this approach requires a strengthened version of the indifference zone probability requirement. It is shown that the same sample size guarantees this requirement as does the earlier one.