Desired Coverage Probability Research Articles

Binomial Confidence Intervals for Rare Events: Importance of Defining Margin of Error Relative to Magnitude of Proportion

Confidence interval performance is typically assessed in terms of two criteria: coverage probability and interval width (or margin of error). In this article, we assess the performance of four common proportion interval estimators: the Wald, Clopper-Pearson (exact), Wilson and Agresti-Coull, in the context of rare-event probabilities. We define the interval precision in terms of a relative margin of error which ensures consistency with the magnitude of the proportion. Thus, confidence interval estimators are assessed in terms of achieving a desired coverage probability whilst simultaneously satisfying the specified relative margin of error. We illustrate the importance of considering both coverage probability and relative margin of error when estimating rare-event proportions, and show that within this framework, all four interval estimators perform somewhat similarly for a given sample size and confidence level. We identify relative margin of error values that result in satisfactory coverage while being conservative in terms of sample size requirements, and hence suggest a range of values that can be adopted in practice. The proposed relative margin of error scheme is evaluated analytically, by simulation, and by application to a number of recent studies from the literature.

Statistical inference for generative adversarial networks and other minimax problems

AbstractThis paper studies generative adversarial networks (GANs) from the perspective of statistical inference. A GAN is a popular machine learning method in which the parameters of two neural networks, a generator and a discriminator, are estimated to solve a particular minimax problem. This minimax problem typically has a multitude of solutions and the focus of this paper are the statistical properties of these solutions. We address two key statistical issues for the generator and discriminator network parameters, consistent estimation and confidence sets. We first show that the set of solutions to the sample GAN problem is a (Hausdorff) consistent estimator of the set of solutions to the corresponding population GAN problem. We then devise a computationally intensive procedure to form confidence sets and show that these sets contain the population GAN solutions with the desired coverage probability. Small numerical experiments and a Monte Carlo study illustrate our results and verify our theoretical findings. We also show that our results apply in general minimax problems that may be nonconvex, nonconcave, and have multiple solutions.

Simultaneous selection and inference for varying coefficients with zero regions: a soft-thresholding approach.

Varying coefficient models have been used to explore dynamic effects in many scientific areas, such as in medicine, finance, and epidemiology. As most existing models ignore the existence of zero regions, we propose a new soft-thresholded varying coefficient model, where the coefficient functions are piecewise smooth with zero regions. Our new modeling approach enables us to perform variable selection, detect the zero regions of selected variables, obtain point estimates of the varying coefficients with zero regions, and construct a new type of sparse confidence intervals that accommodate zero regions. We prove the asymptotic properties of the estimator, based on which we draw statistical inference. Our simulation study reveals that the proposed sparse confidence intervals achieve the desired coverage probability. We apply the proposed method to analyze a large-scale preoperative opioidstudy.

A method to reduce the width of confidence intervals by using a normal scores transformation

SummaryIn stating the results of their research, scientists usually want to publish narrow confidence intervals because they give precise estimates of the effects of interest. In many cases, the researcher would want to use the narrowest interval that maintains the desired coverage probability. In this manuscript, we propose a new method of finding confidence intervals that are often narrower than traditional confidence intervals for any individual parameter in a linear model if the errors are from a skewed distribution or from a long‐tailed symmetric distribution. If the errors are normally distributed, we show that the width of the proposed normal scores confidence interval will not be much greater than the width of the traditional interval. If the dataset includes predictor variables that are uncorrelated or moderately correlated then the confidence intervals will maintain their coverage probability. However, if the covariates are highly correlated, then the coverage probability of the proposed confidence interval may be slightly lower than the nominal value. The procedure is not computationally intensive and an R program is available to determine the normal scores 95% confidence interval. Whenever the covariates are not highly correlated, the normal scores confidence interval is recommended for the analysis of datasets having 50 or more observations.

Measuring repeatability of compositional diet estimates: Anexample using quantitative fatty acid signature analysis.

By measuring the temporal consistency, or repeatability, in the diets of predators, we can gain a better understanding of the degree of individual specialization in resource utilization and implications for predator–prey interactions, population dynamics, and food web structure. To measure repeatability, we require repeated diet estimates of individuals over time, such as those derived from quantitative fatty acid signature analysis (QFASA), a popular diet estimation technique. However, diet estimates are often lengthy compositional vectors with many zeros, as some prey will not be consumed by all individuals, precluding the use of previously proposed measures of repeatability. In this paper, we propose a novel approach for inferring repeatability for multivariate data and, in particular, compositional diet estimates. We extend the commonly used measure of repeatability for univariate data to the multivariate compositional setting by utilizing the mean squares obtained from a nonparametric multivariate analysis of variance, and an appropriate choice of statistical distance. Our measure and its extension are compatible with both balanced and unbalanced data sets. Associated confidence intervals via nonparametric bootstrapping are also developed for the case of QFASA diet estimates that incorporate both sampling error and measurement error, where the latter error arises because the diets of predators are estimated. Simulation study results suggest that for practical levels of repeatability, our methods yield confidence intervals with the desired coverage probability even when the sample size relative to the dimension of the data (i.e., number of prey species eaten) is small. We tested our methods using QFASA diet estimates for free‐ranging Northwest Atlantic grey seals. Given the importance of understanding how predator diets vary over time and space, our method may find broad application to other compositional diet estimates, including those derived from the stomach or fecal contents, and stable isotope analyses.

Robust prediction interval estimation for Gaussian processes by cross-validation method

Probabilistic regression models typically use the Maximum Likelihood Estimation or Cross-Validation to fit parameters. These methods can give an advantage to the solutions that fit observations on average, but they do not pay attention to the coverage and the width of Prediction Intervals. A robust two-step approach is used to address the problem of adjusting and calibrating Prediction Intervals for Gaussian Processes Regression. First, the covariance hyperparameters are determined by a standard Cross-Validation or Maximum Likelihood Estimation method. A Leave-One-Out Coverage Probability is introduced as a metric to adjust the covariance hyperparameters and assess the optimal type II Coverage Probability to a nominal level. Then a relaxation method is applied to choose the hyperparameters that minimize the Wasserstein distance between the Gaussian distribution with the initial hyperparameters (obtained by Cross-Validation or Maximum Likelihood Estimation) and the proposed Gaussian distribution with the hyperparameters that achieve the desired Coverage Probability. The method gives Prediction Intervals with appropriate coverage probabilities and small widths.

Confidence intervals that utilize sparsity

We consider a linear regression model with orthogonal regressors and Gaussian random errors with known variance, in the low‐dimensional setting that the length of the regression parameter vector does not exceed the length of the response vector. We suppose that we have uncertain prior information that sparsity holds, that is, that many of the components of the regression parameter vector are zero. Our aim is to construct confidence intervals for the components of this vector with both the desired coverage probability and attractive expected length properties, particularly when sparsity holds. It is known that for fixed‐width confidence intervals centered on penalized maximum likelihood estimators, such as hard‐thresholding, LASSO and SCAD estimators, the achievement of the desired minimum coverage necessarily results in very unattractive expected length properties. We present confidence intervals, with data‐based widths, which achieve our aim. These confidence intervals dominate the usual confidence intervals with the same coverage probability, whenever the degree of sparsity exceeds a known value.

Generalized confidence interval for an agreement between raters.

Estimation and inference are two key components toward the solution of any statistical problem; however, the inferential issues of statistical assessment of agreement among two or more raters have not been well developed as compared to the development of estimation procedures in this area. The fundamental reason for this gap is the complex expression of the concordance correlation coefficient (CCC) that is frequently used in assessing agreement among raters. Large sample-based statistical tests for CCC often fail to produce desired results for small samples. Hence, inferential procedures for small samples are urgently needed to evaluate agreement between raters. We argue that hypothesis testing of CCC has little value in practice due to the absence of a gold standard of agreement. In this article, we construct the generalized confidence interval (GCI) for CCC utilizing a bivariate normal distribution of measurements, and also develop a large sample-based confidence interval (LSCI). We establish satisfactory performance of GCI by providing the desired coverage probability (CP) via simulation. Results of GCI and LSCI are illustrated and compared with a data set of a recent study performed at U.S. Department of Veterans Affairs, Hines.

Confidence intervals for nonparametric quantile regression: an emphasis on smoothing splines approach

SummaryIn this paper we consider the problem of constructing confidence intervals for nonparametric quantile regression with an emphasis on smoothing splines. The mean‐based approaches for smoothing splines of Wahba (1983) and Nychka (1988) may not be efficient for constructing confidence intervals for the underlying function when the observed data are non‐Gaussian distributed, for instance if they are skewed or heavy‐tailed. This paper proposes a method of constructing confidence intervals for the unknown τth quantile function (0<τ<1) based on smoothing splines. In this paper we investigate the extent to which the proposed estimator provides the desired coverage probability. In addition, an improvement based on a local smoothing parameter that provides more uniform pointwise coverage is developed. The results from numerical studies including a simulation study and real data analysis demonstrate the promising empirical properties of the proposed approach.

MO-FG-CAMPUS-TeP2-04: Optimizing for a Specified Target Coverage Probability

Purpose: The purpose of this work is to develop a method for inverse planning of radiation therapy margins. When using this method the user specifies a desired target coverage probability and the system optimizes to meet the demand without any explicit specification of margins to handle setup uncertainty. Methods: The method determines which voxels to include in an optimization function promoting target coverage in order to achieve a specified target coverage probability. Voxels are selected in a way that retains the correlation between them: The target is displaced according to the setup errors and the voxels to include are selected as the union of the displaced target regions under the x% best scenarios according to some quality measure. The quality measure could depend on the dose to the considered structure alone or could depend on the dose to multiple structures in order to take into account correlation between structures. Results: A target coverage function was applied to the CTV of a prostate case with prescription 78 Gy and compared to conventional planning using a DVH function on the PTV. Planning was performed to achieve 90% probability of CTV coverage. The plan optimized using the coverage probability function had P(D98 > 77.95 Gy) = 0.97 for the CTV. The PTV plan using a constraint on minimum DVH 78 Gy at 90% had P(D98 > 77.95) = 0.44 for the CTV. To match the coverage probability optimization, the DVH volume parameter had to be increased to 97% which resulted in 0.5 Gy higher average dose to the rectum. Conclusion: Optimizing a target coverage probability is an easily used method to find a margin that achieves the desired coverage probability. It can lead to reduced OAR doses at the same coverage probability compared to planning with margins and DVH functions.

Process capability control chart for non-normal data – evidence of on-going capability assessment

Process capability indices are widely used for various quality assurance activities in many high-tech industries. By a suitable analysis of an index, a production department can trace and improve a poor process to enhance the quality level and satisfy customer requirements. Recently, some practical process capability control charts were developed for illustrating the continuous assessments of normal process capability. These charts contain information regarding the ability of the process to meet or exceed customer requirement over the life of a process. Monitoring the behaviour of the process’s ability to meet customer requirement can provide managers with feedback and evidence of the process’s ability in terms they understand. However, process distribution is often ab normal. Therefore, this study incorporates bootstrap procedure to construct capability control chart for non-normal processes. The simulations show that the empirical coverage probability of this chart is close to the desired coverage probability as the number of subgroup is large enough.

Effective Control Chart for Monitoring the Stability of Non-normal Process Capability

Quality has been shown as the major criterion in many supplier selection studies. Among various quality assurance activities, process capability indices are widely used to quantify the relationship between the actual process performance and specification limits in many high-tech industries. By a suitable analysis of an index, a production department can trace and improve a poor process to enhance the quality level and satisfy customer requirements. Recently, some practical process capability control charts were developed for illustrating the continuous assessments of process capability. These charts contain information regarding the ability of the process to meet or exceed customer requirement over the life of a process. However, the process distribution is often ab-normal. Therefore, this study integrates the bootstrap procedure to construct a capability control chart for non-normal processes. The simulations show that the empirical coverage probability of our proposed chart is close to the desired coverage probability as the size of subgroup is large enough.

Statistical methods for thermonuclear reaction rates and nucleosynthesis simulations

Rigorous statistical methods for estimating thermonuclear reaction rates and nucleosynthesis are becoming increasingly established in nuclear astrophysics. The main challenge being faced is that experimental reaction rates are highly complex quantities derived from a multitude of different measured nuclear parameters (e.g., astrophysical S-factors, resonance energies and strengths, particle and γ-ray partial widths). We discuss the application of the Monte Carlo method to two distinct, but related, questions. First, given a set of measured nuclear parameters, how can one best estimate the resulting thermonuclear reaction rates and associated uncertainties? Second, given a set of appropriate reaction rates, how can one best estimate the abundances from nucleosynthesis (i.e., reaction network) calculations? The techniques described here provide probability density functions that can be used to derive statistically meaningful reaction rates and final abundances for any desired coverage probability. Examples are given for applications to s-process neutron sources, core-collapse supernovae, classical novae, and Big Bang nucleosynthesis.

Simultaneous confidence interval for quantile regression

This paper considers a problem of constructing simultaneous confidence intervals for quantile regression. Recently, Krivobokova et al. (J Am Stat Assoc 105:852---863, 2010) provided simultaneous confidence intervals for penalized spline estimator. However, it is well known that the conventional mean-based penalized spline and its confidence intervals collapse when data are not normally distributed such as skewed or heavy-tailed, and hence, the resultant confidence intervals further provide low coverage probability. To overcome this problem, this paper proposes a new approach that constructs simultaneous confidence intervals for penalized quantile spline estimator, which yields a desired coverage probability. The results obtained from numerical experiments and real data validate the effectiveness of the proposed method.

Random Sets and Exact Confidence Regions

An important problem in statistics is the construction of confidence regions for unknown parameters. In most cases, asymptotic distribution theory is used to construct confidence regions, so any coverage probability claims only hold approximately, for large samples. This paper describes a new approach, using random sets, which allows users to construct exact confidence regions without appeal to asymptotic theory. In particular, if the user-specified random set satisfies a certain validity property, confidence regions obtained by thresholding the induced data-dependent plausibility function are shown to have the desired coverage probability.

Confidence intervals for the selected population in randomized trials that adapt the population enrolled

It is a challenge to design randomized trials when it is suspected that a treatment may benefit only certain subsets of the target population. In such situations, trial designs have been proposed that modify the population enrolled based on an interim analysis, in a preplanned manner. For example, if there is early evidence during the trial that the treatment only benefits a certain subset of the population, enrollment may then be restricted to this subset. At the end of such a trial, it is desirable to draw inferences about the selected population. We focus on constructing confidence intervals for the average treatment effect in the selected population. Confidence interval methods that fail to account for the adaptive nature of the design may fail to have the desired coverage probability. We provide a new procedure for constructing confidence intervals having at least 95% coverage probability, uniformly over a large class Q of possible data generating distributions. Our method involves computing the minimum factor c by which a standard confidence interval must be expanded in order to have, asymptotically, at least 95% coverage probability, uniformly over Q. Computing the expansion factor c is not trivial, since it is not a priori clear, for a given decision rule, for which data generating distribution leads to the worst-case coverage probability. We give an algorithm that computes c, and then prove an optimality property for the resulting confidence interval procedure.

Confident estimation for density of a biological population based on line transect sampling

Line transect sampling is a very useful method in survey of wildlife population. Confident interval estimation for density D of a biological population is proposed based on a sequential design. The survey area is occupied by the population whose size is unknown. A stopping rule is proposed by a kernel-based estimator of density function of the perpendicular data at a distance. With this stopping rule, we construct several confidence intervals for D by difference procedures. Some bias reduction techniques are used to modify the confidence intervals. These intervals provide the desired coverage probability as the bandwidth in the stopping rule approaches zero. A simulation study is also given to illustrate the performance of this proposed sequential kernel procedure.

Marcus–Talpaz simultaneous lower confidence bounds for contrasts between treatment and control means

We consider comparisons of several treatments with a common control when it is believed that the treatments are at least as effective as the control. In this setting, which is called the tree order, the classical work by Dunnett (1955) is well known. Dunnett proposed a multiple-comparison procedure for treatment means vs. the control mean, which could be inverted to obtain simultaneous lower confidence bounds (SLCBs). Under the same setting, Marcus and Talpaz (1992) proposed a test of homogeneity and SLCBs for contrasts of means. These SLCBs, however, assumed that the contrast coefficients were tree ordered but that the means were unordered. Using the Marcus–Talpaz test of homogeneity, we develop another set of SLCBs, which does utilize the fact that the means are tree ordered and has the desired coverage probability. Using Monte Carlo simulations, we compare Dunnett’s SLCBs and the newly developed SLCBs for equal sample sizes. We examine the two sets of SLCBs in terms of their abilities to detect true mean differences and their relative efficiencies. The results reveal that Dunnett’s SLCBs perform better in the interior of the tree-order cone. When the entire cone, including the boundary, is considered, the new SLCBs perform more favorably. A numerical example is given.

Empirical Bayes Confidence Intervals for Means of Natural Exponential Family-Quadratic Variance Function Distributions with Application to Small Area Estimation

The paper develops empirical Bayes (EB) confidence intervals for population means with distributions belonging to the natural exponential family-quadratic variance function (NEF-QVF) family when the sample size for a particular population is moderate or large. The basis for such development is to find an interval centred around the posterior mean which meets the target coverage probability asymptotically, and then show that the difference between the coverage probabilities of the Bayes and EB intervals is negligible up to a certain order. The approach taken is Edgeworth expansion so that the sample sizes from the different populations need not be significantly large. The proposed intervals meet the target coverage probabilities asymptotically, and are easy to construct. We illustrate use of these intervals in the context of small area estimation both through real and simulated data. The proposed intervals are different from the bootstrap intervals. The latter can be applied quite generally, but the order of accuracy of these intervals in meeting the desired coverage probability is unknown.

Repeated confidence intervals for adaptive group sequential trials

This paper proposes a method for computing conservative confidence intervals for a group sequential test in which an adaptive design change is made one or more times over the course of the trial. The key idea, due to Müller and Schäfer (Biometrics 2001; 57:886-891), is that by preserving the null conditional rejection probability of the remainder of the trial at the time of each adaptive change, the overall type I error rate, taken unconditionally over all possible design modifications, is also preserved. We show how this principle may be extended to construct one-sided confidence intervals by applying the idea to a sequence of dual tests derived from the repeated confidence intervals (RCIs) proposed by Jennison and Turnbull (J. Roy. Statist. Soc. B 1989; 51:301-361). These adaptive RCIs, such as their classical counterparts, have the advantage that they preserve the desired coverage probability even if the pre-specified stopping rule is over-ruled. The statistical methodology is explored by simulations and is illustrated by an application to a clinical trial of deep brain stimulation for Parkinson's disease.