Adaptive Sequential Procedure Research Articles

Adaptive Sequential Procedures for Quantile Selection with Importance Sampling

In this paper, we consider the problem of identifying the simulated system configuration with optimal quantile performance among a finite set of alternative systems. We first propose adaptive sequential selection procedures via direct simulation, where the point and variance estimators for quantiles are sequentially updated. For the sake of extreme quantiles, we develop adaptive quantile selection procedures using importance sampling. Our procedures apply sample average approximation and stochastic approximation for iteratively computing the optimal importance-sampling parameter and the importance-sampling quantile estimators. The statistical validity of the proposed selection procedures is proven in the asymptotic regime. We particularly investigate the consistency and the central limit convergence using both sample average and stochastic approximation for quantile estimation during the adaptive procedures. Empirical results and an illustrative example show the efficiency of the developed procedures.

Distributed sequential estimation procedures

AbstractData collected from distributed sources or sites commonly have different distributions or contaminated observations. Active learning procedures allow us to assess data when recruiting new data into model building. Thus, combining several active learning procedures together is a promising idea, even when the collected data set is contaminated. Here, we study how to conduct and integrate several adaptive sequential procedures at a time to produce a valid result via several machines or a parallel‐computing framework. To avoid distraction by complicated modelling processes, we use confidence set estimation for linear models to illustrate the proposed method and discuss the approach's statistical properties. We then evaluate its performance using both synthetic and real data. We have implemented our method using Python and made it available through Github at https://github.com/zhuojianc/dsep.

A Curtailed Procedure for Selecting Among Treatments With Two Bernoulli Endpoints

This paper is concerned with a closed adaptive sequential procedure for selecting a random-size subset containing experimental treatments that are better than a standard. All the k treatments under considerations are measured by two endpoints accounting for treatment efficacy and treatment safety respectively. The selection is made with regard to the two binary endpoints. An experimental treatment is considered to be better than the standard if its both endpoints have successful rates higher than the standard ones. We provide a step-by-step sampling rule, stopping rule, and decision rule for the proposed procedure. We show that the proposed sequential procedure achieves the same requirements for the probability of a correct selection as does the fixed-sample-size procedure, but requires fewer observations. We use simulations to evaluate the sample size savings of the proposed procedure over the corresponding fixed-sample-size procedure.

Distributions of stopping times in some sequential estimation procedures

A class of sequential estimation procedures is considered in the case when relevant data may become available only at random times. The exact distributions of the optimal stopping time and the number of observations at the moment of stopping are derived in some sequential procedures. The results obtained in an explicit form are applied to derive the expected time of observing the process, the average number of observations and the expected loss of sequential estimation procedures based on delayed observations. The use of the results is illustrated in a special model of normally distributed observations and the Weibull distributed lifetimes. The probabilistic characteristics are also derived for an adaptive sequential procedures and the behavior of the adaptive procedure is compared with the corresponding optimal sequential procedure.

Statistical aspects of the TNK-S2B trial of tenecteplase versus alteplase in acute ischemic stroke: an efficient, dose-adaptive, seamless phase II/III design

Background TNK-S2B, an innovative, randomized, seamless phase II/III trial of tenecteplase versus rt-PA for acute ischemic stroke, terminated for slow enrollment before regulatory approval of use of phase II patients in phase III.Purpose (1) To review the trial design and comprehensive type I error rate simulations and (2) to discuss issues raised during regulatory review, to facilitate future approval of similar designs.Methods In phase II, an early (24-h) outcome and adaptive sequential procedure selected one of three tenecteplase doses for phase III comparison with rt-PA. Decision rules comparing this dose to rt-PA would cause stopping for futility at phase II end, or continuation to phase III. Phase III incorporated two co-primary hypotheses, allowing for a treatment effect at either end of the trichotomized Rankin scale. Assuming no early termination, four interim analyses and one final analysis of 1908 patients provided an experiment-wise type I error rate of <0.05.Results Over 1,000 distribution scenarios, each involving 40,000 replications, the maximum type I error in phase III was 0.038. Inflation from the dose selection was more than offset by the one-half continuity correction in the test statistics. Inflation from repeated interim analyses was more than offset by the reduction from the clinical stopping rules for futility at the first interim analysis.Limitations Design complexity and evolving regulatory requirements lengthened the review process.Conclusions (1) The design was innovative and efficient. Per protocol, type I error was well controlled for the co-primary phase III hypothesis tests, and experiment-wise. (2a) Time must be allowed for communications with regulatory reviewers from first design stages. (2b) Adequate type I error control must be demonstrated. (2c) Greater clarity is needed on (i) whether this includes demonstration of type I error control if the protocol is violated and (ii) whether simulations of type I error control are acceptable. (2d) Regulatory agency concerns that protocols for futility stopping may not be followed may be allayed by submitting interim analysis results to them as these analyses occur.

Editorial

Editorial

Sequential design in quality control and validation of land cover databases

AbstractWe have faced the problem of evaluating the quality of land cover databases produced through photo‐interpretation of remote‐sensing data according to a legend of land cover types. First, we have considered the quality control, that is, the comparison of a land cover database with the result of the photo‐interpretation made by a more expert photo‐interpreter, on a sample of the polygons. Then we have analysed the problem of validation, that is, the check of the photo‐interpretation through a ground survey.We have used the percentage of area correctly photo‐interpreted as a quality measure. Since the kind of land cover type and the size of the polygons affect the probability of making mistakes in the photo‐interpretation, we stratify the polygons according to two variables: the land cover type of the photo‐interpretation and the size of the polygons.We have proposed an adaptive sequential procedure with permanent random numbers in which the sample size per stratum is dependent on the previously selected units but the sample selection is not, and the stopping rule is not based on the estimates of the quality parameter. We have proved that this quality control and validation procedure allows unbiased and efficient estimates of the quality parameters and allows reaching high precision of estimates with the smallest sample size. Copyright © 2009 John Wiley &amp; Sons, Ltd.

An Adaptive Sequential Procedure for Efficient Optimization of the Sheet Metal Spinning Process

AbstractDue to technological advances in science, the complexity of processes under investigation increased continuously in recent years. For example, in mechanical engineering, there are often highly nonlinear input–output relationships combined with a large number of constraints. Sheet metal spinning is one example of such a production process. Many flexible and powerful methods to global optimization have been developed. In this paper, a sequential approach originally developed for computer experiments will be adopted and applied to optimize the spinning process based on physical experiments. This approach sequentially refines the model by adding new design points based on the expected improvement criterion. This criterion balances the need to observe at the predicted optimum with the need to investigate the design space in areas of high uncertainty. However to guarantee an efficient optimization of the spinning process, this approach has to be embedded in a more substantial procedure. One reason for this is the liability of workpiece failure for most of the operable design space. Since the shape of the failure region is unknown, many missing observations have to be expected when exploring the design space. The other reason is the need to incorporate available process knowledge of sheet metal spinning to improve the efficiency in optimization. The main problem in implementing this information is a changing process behavior for different geometries and materials used. Hence, if a component with a new geometry has to be optimized, it is difficult to include available process knowledge. In this paper, an adaptive sequential optimization procedure (ASOP) is presented to cope with these problems in order to guarantee an efficient optimization of such complex processes. The approach is exemplified by optimizing the spinning process for a fixed geometry. Copyright © 2005 John Wiley &amp; Sons, Ltd.

A note on two-stage and sequential fixed-width intervals for the parameter in the uniform density

Two solutions based on two-stage sampling are given to the problem of obtaining a confidence interval having a specified width 2 d for the parameter of the uniform (0,θ) density. When d/0 is small, they yield smaller sample sizes than that of Graybill and Connell (1964). An adaptive sequential procedure is also developed and compared with the two-stage procedures.

Closed adaptive sequential paired-comparison selection procedures

Closed adaptive sequential procedures were introduced by Bechhofer and Kulkarni (1982) as a new method for selecting the best of t ( 2) Bernoulli populations. We modify this approach to obtain two procedures for selecting the best of t objects in a curtailed Round Robin-type paired-comparison experiment. Objects are paired sequentially and the experiment is stopped as soon as one object has achieved a number of preferences that no other object can equal (weak curtailment) or surpass (strong curtailment) if the tournament were run to completion. Ties for first place are broken at random, Weak curtailment clearly selects the same object as the complete Round Robin, with generally substantially fewer comparisons. Strong curtailment effects a further appreciable reduction in the average number of comparisons needed. It is shown that the probabilities of selecting a particular object are the same for weak and strong curtailment if the Bradley-Terry model holds, but generally not otherwise. Some comparisons are...

Optimal Properties of the Bechhofer-Kulkarni Bernoulli Selection Procedure

In a recent article Bechhofer and Kulkarni proposed a class of closed adaptive sequential procedures for selecting that one of $k \geq 2$ Bernoulli populations with the largest single-trial success probability. These sequential procedures which take no more than $n$ observations from any one of the $k$ populations achieve the same probability of a correct selection as does a single-stage procedure which takes exactly $n$ observations from every one of the $k$ populations. In addition, they often require substantially less than a total of $kn$ observations to terminate sampling. Amongst other problems, Bechhofer and Kulkarni considered the problem of devising a procedure within this class which minimizes the expected total number of observations to terminate sampling. For their proposed procedure they cited several optimality properties for the case $k = 2$ and conjectured additional optimality properties for the case $k \geq 3$. In this article we use a new method of proof to establish stronger results than those cited by Bechhofer and Kulkarni for the case $k = 2$, and prove stronger results than those conjectured for $k \geq 3$. We also describe a new procedure for $k \geq 3$ and prove that it minimizes the expected total number of observations to terminate sampling when all of the success probabilities are small.

Optimal Bayes procedures for selecting the better of two Bernoulli populations

Let Π 1 and Π 2 be two Bernoulli populations with probabilities of success p 1 and p 2, respectively. In a recent paper Bechhofer and Kulkarni (1982) proposed a class of closed adaptive sequential procedures for the problem of selecting the population with the larger success probability. The procedures in this class achieve the same probability of correct selection, uniformly in ( p 1, p 2), as does the single stage procedure of Sobel and Huyett (1957) which takes exactly n observations from each population. This paper considers the problem of selecting an optimal procedure within this class which minimizes the expected total number of observations under the assumption that p 1 and p 2 have independent prior Beta distributions. Recursion equations are derived which can be used to determine such an optimal procedure. Empirical procedures are described which are shown numerically to perform very well when compared with the optimal procedures. The class of empirical procedures for symmetric Beta priors is conjectured to be minimax within the class of procedures considered by Bechhofer and Kulkarni; strong arguments are presented to support this conjecture. The empirical procedure using uniform prior distributions for p 1 and p 2 is shown to have very desirable properties. The performance characteristics of this procedure are studied in detail, and based on these it is proposed as an attractive procedure to use in practice.

An optimal sequential procedure for selecting the best bernoulli process—a review

AbstractThis article describes a new closed adaptive sequential procedure proposed by Bechhofer and Kulkarni for selecting the Bernoulli population which has the largest success probability. It can be used effectively for selecting the production process with the largest proportion of conforming items, and thus is applicable in vendor selection situations. The performance of this procedure is compared to that of the Sobel‐Huyett single‐stage procedure, and to a curtailed version of the single‐stage procedure, all of which guarantee the same probability of a correct selection. Optimal properties of the Bechhofer‐Kulkarni procedure are stated; quantitative assessments of important performance characteristics of the procedure are given. These demonstrate conclusively the superiority of the new procedure over that of the competing procedures. Relevant areas of application (including clinical trials) are described. Appropriate literature references are provided.

Adaptive sequential procedures for comparing new treatments with a standard

In this paper we propose and study two sequential elimination procedures for selecting all new treatments better than a standard or control treatment. These procedures differ from those previously proposed in that we assume variances are unequal and unknown. Expressions for asymptotic expected sample sizes are given. Confidence intervals associated with the procedures are also discussed.

A monte carlo study of the performance of a closed adaptive sequential procedure for selecting the best bernoulli population

The performance characteristics of a closed adaptive sequential procedure for selecting the best Bernoulli population, recently proposed by Bechhofer and Kulkarni, are studied using Monte Carlo simulation. The sequential procedure has been shown to achieve the same probability of a correct selection, uniformly in the unknown single-trial success probabilities p i (1≦i≦k) as does the corresponding single-stage procedure which takes exactly n observations from each of the k populations; in addition, has been shown to possess severa! highly desirable optimal properties. In the present paper we obtain precise MC estimates of E{N(i)} (1≦i≦k) and E{N) where N (i) is the total number of observations taker. from the population associated with p (i) (1≦i≦k) when terminates sampling, and here are the ordered p i (1≦i≦k) MC estimates of other related performance characteristics are also obtained. Special emphasis is given to k = 4 and 5 populations for which no results had been obtained heretofore because of the c...

On the performance characteristics of a closed adaptive sequential procedure for selecting the best bernoulli population

In a recent paper, Bechhofer and Kulkarni proposed closed adaptive sequential procedures for a general class of k—population Bernoulli selection goals. These sequential selection procedures achieve the same probability of a correct selection, uniformly in the unknown single—trail success probabilities , as do the corresponding single—stage selection procedures which take exactly n observations from each of the k populations. The sequential procedures always require less (often substantially less) than kn observations to terminate experimentation. This earlier paper described the procedures, discussed their performance in general terms, and cited several of their optimality properties. In the present paper we specialize these procedures, and focus on the particular goal of selecting the population associated with p[k] where are the ordered .Stronger optimality results than those given in our earlier paper are cited. We give exact numerical results for such performance characteristics of the sequential pr...

Adaptive sequential procedures for selecting the best of several normal populations†

There are k(≧2) competing normal populations with common known variance and unknown means . Let denote the ordered values of the {z i}. Nothing is known concerning the pairing of the {z.Θ i} and the {z.Θ [i]}. In the location invariant identification problem, the differences are given and it is desired to select the population associated with {z.Θ [k]}. Of particular interest is the slippage configuration where δ *>0 is given. We restrict attention to procedures that guarantee that the population with the largest mean is correctly selected with probability at least P * where is preassigned. Essentially, this requirement is satisfied by the stopping rule of Bechhofer, Kiefer and Sobel (Sequential Identification and Ranking Procedures, Univ. of Chicago Press 1968, Chap. 3) independently of the sampling rule and thus data dependent allocation rules can be considered. Unlike the case k=2 (see Robbins and Siegmund ( 1974) J. Am. Statist. Assoc. 69, 132-139), when k≧3,substantial savings in expected tot...

Frequency-difference thresholds in Chinchilla using conditioned avoidance techniques

Chinchillas were trained to discriminate or detect pure-tone frequency differences using shock-avoidance procedures. Two psycho-physical paradigms were investigated. Initial attempts at determining frequency-difference thresholds utilized a complex discrimination paradigm. Although discriminations of large frequency differences were obtained from Chinchilla with adequate success, discriminations of very small frequency differences could not be trained. Therefore, determinations of frequency-difference thresholds were not possible with this complex discrimination paradigm. A second relatively simple detection paradigm, involving the detection of frequency alternation in an on-going train of tone bursts, proved to be a somewhat more successful technique. Frequency alternation detection was quickly learned by the Chinchillas and frequency-difference thresholds were obtained with an adaptive sequential procedure. Those frequency-difference thresholds are comparable to frequency-difference thresholds obtained in other animal species by previous investigators. Some of the difficulties in maintaining reliable behavior in Chinchillas with shock avoidance techniques are discussed. [Work supported by DRF grant.]

Signal Ensemble and Intensive Discrimination

Intensive DLs were obtained for three listeners as a function of carrier level for four signal ensembles. The intensive difference signal was defined either by amplitude modulation of the carrier by an 8-Hz sinusoid or by addition of a 500-msec intensive increment to the carrier. Carrier level was defined either by a continuous tone or by a 500-msec pulsed tone at 1000 Hz. An adaptive sequential procedure, incorporating 4IFC method, was used in estimating thresholds. All except the steady-state, pulsed-carrier ensemble showed the improvement in discrimination usually associated with ΔI/I. DLs for the continuous carrier tended to be smaller than those for pulsed carrier. Also, detection was more acute for the AM signal than for the fixed difference in intensity, except near threshold where this relationship reversed itself. The atypical magnitude and level dependence of DLs for the pulsed-carrier steady-state ensemble can be related to other aspects of auditory perception.

Frequency Discrimination in Noise

Frequency DLs were obtained near masked threshold (1–15 dB SL) at 250, 1000, and 4000 Hz. Three spectrum levels (31.5, 51.5, and 71.5 dB) were used. An adaptive sequential procedure based on 4IFC was used in estimating ΔF. Also, two signal modes were used. In one, the pulsed tone presented during the “signal interval” was frequency modulated. In the other, the pulsed tone presented during the “signal interval” was a steady, but different, frequency. Characteristic improvement was seen at all frequencies when ΔF was plotted as a function of sensation level. At 4000 Hz, however, a significant “noise-level” effect was noted, in which discrimination at the 51.5-dB spectrum level (band level of 86 dB) was poorer than that for 31.5 and 71.5 dB. This “noise-level” effect was essentially the same for both modes of signal presentation. While this effect is difficult to explain in psychophysical terms, the literature on physiological acoustics offers some promise of accounting for it.