Coverage Properties Of Confidence Intervals Research Articles

The empirical coverage of confidence intervals: Point estimates and confidence intervals for confidence levels

Many confidence intervals calculated in practice are potentially not exact, either because the requirements for the interval estimator to be exact are known to be violated, or because the (exact) distribution of the data is unknown. If a confidence interval is approximate, the crucial question is how well its true coverage probability approximates its intended coverage probability. In this paper we propose to use the bootstrap to calculate an empirical estimate for the (true) coverage probability of a confidence interval. In the first instance, the empirical coverage can be used to assess whether a given type of confidence interval is adequate for the data at hand. More generally, when planning the statistical analysis of future trials based on existing data pools, the empirical coverage can be used to study the coverage properties of confidence intervals as a function of type of data, sample size, and analysis scale, and thus inform the statistical analysis plan for the future trial. In this sense, the paper proposes an alternative to the problematic pretest of the data for normality, followed by selection of the analysis method based on the results of the pretest. We apply the methodology to a data pool of bioequivalence studies, and in the selection of covariance patterns for repeated measures data.

Coverage Properties of Confidence Intervals for Generalized Additive Model Components

Abstract. We study the coverage properties of Bayesian confidence intervals for the smooth component functions of generalized additive models (GAMs) represented using any penalized regression spline approach. The intervals are the usual generalization of the intervals first proposed by Wahba and Silverman in 1983 and 1985, respectively, to the GAM component context. We present simulation evidence showing these intervals have close to nominal ‘across‐the‐function’ frequentist coverage probabilities, except when the truth is close to a straight line/plane function. We extend the argument introduced byNychka in 1988for univariate smoothing splines to explain these results. The theoretical argument suggests that close to nominal coverage probabilities can be achieved, provided that heavy oversmoothing is avoided, so that the bias is not too large a proportion of the sampling variability. The theoretical results allow us to derive alternative intervals from a purely frequentist point of view, and to explain the impact that the neglect of smoothing parameter variability has on confidence interval performance. They also suggest switching the target of inference for component‐wise intervals away from smooth components in the space of the GAM identifiability constraints.

Likelihood-Based Inferences under Isolation by Distance: Two-Dimensional Habitats and Confidence Intervals

Likelihood-based methods of inference of population parameters from genetic data in structured populations have been implemented but still little tested in large networks of populations. In this work, a previous software implementation of inference in linear habitats is extended to two-dimensional habitats, and the coverage properties of confidence intervals are analyzed in both cases. Both standard likelihood and an efficient approximation are considered. The effects of misspecification of mutation model and dispersal distribution, and of spatial binning of samples, are considered. In the absence of model misspecification, the estimators have low bias, low mean square error, and the coverage properties of confidence intervals are consistent with theoretical expectations. Inferences of dispersal parameters and of the mutation rate are sensitive to misspecification or to approximations inherent to the coalescent algorithms used. In particular, coalescent approximations are not appropriate to infer the shape of the dispersal distribution. However, inferences of the neighborhood parameter (or of the product of population density and mean square dispersal rate) are generally robust with respect to complicating factors, such as misspecification of the mutation process and of the shape of the dispersal distribution, and with respect to spatial binning of samples. Likelihood inferences appear feasible in moderately sized networks of populations (up to 400 populations in this work), and they are more efficient than previous moment-based spatial regression method in realistic conditions.

Mixed models for data from thorough QT studies: part 1. assessment of marginal QT prolongation

We investigate mixed models for repeated measures data from cross-over studies in general, but in particular for data from thorough QT studies. We extend both the conventional random effects model and the saturated covariance model for univariate cross-over data to repeated measures cross-over (RMC) data; the resulting models we call the RMC model and Saturated model, respectively. Furthermore, we consider a random effects model for repeated measures cross-over data previously proposed in the literature. We assess the standard errors of point estimates and the coverage properties of confidence intervals for treatment contrasts under the various models. Our findings suggest: (i) Point estimates of treatment contrasts from all models considered are similar; (ii) Confidence intervals for treatment contrasts under the random effects model previously proposed in the literature do not have adequate coverage properties; the model therefore cannot be recommended for analysis of marginal QT prolongation; (iii) The RMC model and the Saturated model have similar precision and coverage properties; both models are suitable for assessment of marginal QT prolongation; and (iv) The Akaike Information Criterion (AIC) is not a reliable criterion for selecting a covariance model for RMC data in the following sense: the model with the smallest AIC is not necessarily associated with the highest precision for the treatment contrasts, even if the model with the smallest AIC value is also the most parsimonious model.

A simulation model for designing groundfish trawl surveys

This paper describes a convenient simulation model, based on the compound binomial-gamma distribution, to assist the planning and design of groundfish trawl surveys. The analysis uses swept-area density measurements from stratified tows to give a simple nonparametric biomass estimate. A parametric simulation model requires only three input parameters for each stratum, which can be estimated initially from past surveys or commercial fishery data. Analytical results provide intuitive algorithms for estimating variances, investigating tow allocation strategies, and exploring potential survey results. Simulations make it possible to compare the estimated biomass with its true value and to assess coverage properties of confidence intervals obtained from bootstraps. Bias correction and acceleration both improve the results, but small samples taken from populations with highly variable densities tend to produce underestimates of available biomass. The simulation framework allows easy adaptation to address broader issues, such as the design of a multispecies survey.

Estimating the responsiveness of an instrument using more than two repeated measures.

The responsiveness of a measuring instrument is its ability to detect change over time. A commonly used index of responsiveness is the effect size for paired differences. This paper generalizes the effect size for paired differences to more than two repeated observations per subject. The sampling distribution of the generalized responsiveness statistic, Rt, is simulated for a range of plausible parameter values and for a range of sample sizes varying both the number of subjects (n) and the number of observations per subject (t). The coverage properties of confidence intervals constructed by four methods are compared. Confidence intervals based on jackknife estimates of the standard error and bias of Rt have good coverage properties even when n and t are small. The methods are used to determine which of two standard quality-of-life measures is more responsive to improvements in quality of life following surgery for early-stage breast cancer.

Bayesian Versus Frequentist Measures of Error in Small Area Estimation

Summary This paper compares analytically and empirically the frequentist and Bayesian measures of error in small area estimation. The model postulated is the nested error regression model which allows for random small area effects to represent the joint effect of small area characteristics that are not accounted for by the fixed regressor variables. When the variance components are known, then, under a uniform prior for the regression coefficients and normality of the error terms, the frequentist and the Bayesian approaches yield the same predictors and prediction mean-squared errors (MSEs) (defined accordingly). When the variance components are unknown, it is common practice to replace the unknown variances by sample estimates in the expressions for the optimal predictors, so that the resulting empirical predictors remain the same under the two approaches. The use of this paradigm requires, however, modifications to the expressions of the prediction MSE to account for the extra variability induced by the need to estimate the variance components. The main focus of this paper is to review and compare the modifications to prediction MSEs proposed in the literature under the two approaches, with special emphasis on the orders of the bias of the resulting approximations to the true MSEs. Some new approximations based on Monte Carlo simulation are also proposed and compared with the existing methods. The advantage of these approximations is their simplicity and generality. Finite sample frequentist properties of the various methods are explored by a simulation study. The main conclusions of this study are that the use of second-order bias corrections generally yields better results in terms of the bias of the MSE approximations and the coverage properties of confidence intervals for the small area means. The Bayesian methods are found to have good frequentist properties, but they can be inferior to the frequentist methods. The second-order approximations under both approaches have, however, larger variances than the corresponding first-order approximations which in most cases result in higher MSEs of the MSE approximations.