Coverage Probabilities Of Confidence Intervals Research Articles

Drug–Drug Interaction Prediction Assessment

Model-based drug–drug interaction (DDI) is an important in-silico tool to assess the in vivo consequences of in vitro DDI. Before its general application to new drug compounds, the DDI model is always established from known interaction data. For the first time, tests for difference and equivalent tests are implemented to compare reported and model-base simulated DDI (log AUCR) in the sample mean and variance. The biases and predictive confidence interval coverage probabilities are introduced to assess the DDI prediction performance. Sample size and power guidelines are developed for DDI model simulations. These issues have never been discussed in trial simulation studies to investigate DDI prediction. A ketoconazole (KETO)/midazolam (MDZ) example is employed to demonstrate these statistical methods. Based on published KETO and MDZ pharmacokinetics data and their in vitro inhibition rate constant data, current model-based DDI prediction underpredicts the area under concentration curve ratio (AUCR) and its between-subject variance compared to the reported study.

Optimal estimators of the position of a mass extinction when recovery potential is uniform

Numerous methods have been developed to estimate the position of a mass extinction boundary while accounting for the incompleteness of the fossil record. Here we describe the point estimator and confidence interval for the extinction that are optimal under the assumption of uniform preservation and recovery potential, and independence among taxa. First, one should pool the data from all taxa into one combined “supersample.” Next, one can then apply methods proposed by Strauss and Sadler (1989) for a single taxon. This gives the optimal point estimator in the sense that it has the smallest variance among all possible unbiased estimators. The corresponding confidence interval is optimal in the sense that it has the shortest average width among all possible intervals that are invariant to measurement scale. These optimality properties hold even among methods that have not yet been discovered. Using simulations, we show that the optimal estimators substantially improve upon the performance of other existing methods. Because the assumptions of uniform recovery and independence among taxa are strong ones, it is important to assess to what extent they are satisfied by the data. We demonstrate the use of probability plots for this purpose. Finally, we use simulations to explore the sensitivity of the optimal point estimator and confidence interval to nonuniformity and lack of independence, and we compare their performance under these conditions with existing methods. We find that nonuniformity strongly biases the point estimators for all methods studied, inflates their standard errors, and degrades the coverage probabilities of confidence intervals. Lack of independence has less effect on the accuracy of point estimates as long as recovery potential is uniform, but it, too, inflates the standard errors and degrades confidence interval coverage probabilities.

Weighted analyses for cohort sampling designs

Weighted analysis methods are considered for cohort sampling designs that allow subsampling of both cases and non-cases, but with cases generally sampled more intensively. The methods fit into the general framework for the analysis of survey sampling designs considered by Lin (Biometrika 87:37-47, 2000). Details are given for applying the general methodology in this setting. In addition to considering proportional hazards regression, methods for evaluating the representativeness of the sample and for estimating event-free probabilities are given. In a small simulation study, the one-sample cumulative hazard estimator and its variance estimator were found to be nearly unbiased, but the true coverage probabilities of confidence intervals computed from these sometimes deviated significantly from the nominal levels. Methods for cross-validation and for bootstrap resampling, which take into account the dependencies in the sample, are also considered.

A Bayesian Spatial Multimarker Genetic Random‐Effect Model for Fine‐Scale Mapping

Multiple markers in linkage disequilibrium (LD) are usually used to localize the disease gene location. These markers may contribute to the disease etiology simultaneously. In contrast to the single-locus tests, we propose a genetic random effects model that accounts for the dependence between loci via their spatial structures. In this model, the locus-specific random effects measure not only the genetic disease risk, but also the correlations between markers. In other words, the model incorporates this relation in both mean and covariance structures, and the variance components play important roles. We consider two different settings for the spatial relations. The first is our proposal, relative distance function (RDF), which is intuitive in the sense that markers nearby are likely to correlate with each other. The second setting is a common exponential decay function (EDF). Under each setting, the inference of the genetic parameters is fully Bayesian with Markov chain Monte Carlo (MCMC) sampling. We demonstrate the validity and the utility of the proposed approach with two real datasets and simulation studies. The analyses show that the proposed model with either one of two spatial correlations performs better as compared with the single locus analysis. In addition, under the RDF model, a more precise estimate for the disease locus can be obtained even when the candidate markers are fairly dense. In all simulations, the inference under the true model provides unbiased estimates of the genetic parameters, and the model with the spatial correlation structure does lead to greater confidence interval coverage probabilities.

The analysis of multivariate recurrent events with partially missing event types

In many clinical studies, subjects are at risk of experiencing more than one type of potentially recurrent event. In some situations, however, the occurrence of an event is observed, but the specific type is not determined. We consider the analysis of this type of incomplete data when the objectives are to summarize features of conditional intensity functions and associated treatment effects, and to study the association between different types of event. Here we describe a likelihood approach based on joint models for the multi-type recurrent events where parameter estimation is obtained from a Monte-Carlo EM algorithm. Simulation studies show that the proposed method gives unbiased estimators for regression coefficients and variance-covariance parameters, and the coverage probabilities of confidence intervals for regression coefficients are close to the nominal level. When the distribution of the frailty variable is misspecified, the method still provides estimators of the regression coefficients with good properties. The proposed method is applied to a motivating data set from an asthma study in which exacerbations were to be sub-typed by cellular analysis of sputum samples as eosinophilic or non-eosinophilic.

Comparison of the Andersen–Gill model with poisson and negative binomial regression on recurrent event data

Many generalizations of the Cox proportional hazard method have been elaborated to analyse recurrent event data. The Andersen–Gill model was proposed to handle event data following Poisson processes. This method is compared with non-survival approaches, such as Poisson and negative binomial regression. The comparison is performed on data simulated according to various event-generating processes and differing in subject heterogeneity. When robust standard error estimates are applied, for Poisson processes the Andersen–Gill approach is comparable to a negative binomial regression, whereas the poisson regression has comparable coverage probabilities of confidence intervals, but increased type I error rates; however, none of the methods can generate unbiased parameter estimates with data violating the independent increment assumption. These findings are illustrated by data from a clinical trial of the efficacy of a new pneumococcal vaccine for prevention of otitis media.

Confidence intervals for candidate gene effects and environmental factors in population‐based association studies of families

Complex diseases are influenced by both genetic and environmental factors. Studies of individuals or of families can be used to examine the association of genetic factors, such as candidate genes, and other risk factors with the presence or absence of complex disorders. If families are investigated, whether or not they are randomly ascertained, possible familial correlation among observations must be considered. We have compared two statistical approaches for analyzing correlated binary data from randomly ascertained nuclear families. The generalized estimating equations approach (GEE) can be used to adjust for familial correlation. The relationship between covariates and the response is modelled, and the correlations among family members are treated as nuisance parameters. For comparison, we have proposed two strategies from a hierarchical nonparametric bootstrap approach. One strategy (S1) samples family units, preserving the structure and correlation within each family. A second and novel strategy (S2) also samples family units but then randomly samples offspring with replacement in each family. We applied the methods to data from a study of cardiovascular disease, and followed up with a simulation study in which family data were generated from an underlying multifactorial genetic model. Although the bootstrap approach was more computationally demanding, it outperformed the GEE in terms of confidence interval coverage probabilities for all sample sizes considered.

Assessing confidence intervals for the tail index by Edgeworth expansions for the Hill estimator

We establish Edgeworth expansions for the distribution function of the standardized Hill estimator for the reciprocal of the index of regular variation of the tail of a distribution function. The expansions are used to derive expansions for coverage probabilities of confidence intervals for the tail index based on the Hill estimator.

Appropriateness of some resampling‐based inference procedures for assessing performance of prognostic classifiers derived from microarray data

The goal of many gene-expression microarray profiling clinical studies is to develop a multivariate classifier to predict patient disease outcome from a gene-expression profile measured on some biological specimen from the patient. Often some preliminary validation of the predictive power of a profile-based classifier is carried out using the same data set that was used to derive the classifier. Techniques such as cross-validation or bootstrapping can be used in this setting to assess predictive power, and if applied correctly, can result in a less biased estimate of predictive accuracy of a classifier. However, some investigators have attempted to apply standard statistical inference procedures to assess the statistical significance of associations between true and cross-validated predicted outcomes. We demonstrate in this paper that naïve application of standard statistical inference procedures to these measures of association under null situations can result in greatly inflated testing type I error rates. Under alternatives of small to moderate associations, confidence interval coverage probabilities may be too low, although for very large associations coverage probabilities approach their intended values. Our results suggest that caution should be exercised in interpreting some of the claims of exceptional prognostic classifier performance that have been reported in prominent biomedical journals in the past few years.

On the Large-Sample Minimal Coverage Probability of Confidence Intervals After Model Selection

We give a large-sample analysis of the minimal coverage probability of the usual confidence intervals for regression parameters when the underlying model is chosen by a “conservative” (or “overconsistent”) model selection procedure. We derive an upper bound for the large-sample limit minimal coverage probability of such intervals that applies to a large class of model selection procedures including the Akaike information criterion as well as various pretesting procedures. This upper bound can be used as a safeguard to identify situations where the actual coverage probability can be far below the nominal level. We illustrate that the (asymptotic) upper bound can be statistically meaningful even in rather small samples.

ON THE COVERAGE PROBABILITY OF CONFIDENCE INTERVALS IN REGRESSION AFTER VARIABLE SELECTION

SummaryThis paper considers a linear regression model with regression parameter vector β. The parameter of interest is θ= aTβ where a is specified. When, as a first step, a data‐based variable selection (e.g. minimum Akaike information criterion) is used to select a model, it is common statistical practice to then carry out inference about θ, using the same data, based on the (false) assumption that the selected model had been provided a priori. The paper considers a confidence interval for θ with nominal coverage 1 ‐ α constructed on this (false) assumption, and calls this the naive 1 ‐ α confidence interval. The minimum coverage probability of this confidence interval can be calculated for simple variable selection procedures involving only a single variable. However, the kinds of variable selection procedures used in practice are typically much more complicated. For the real‐life data presented in this paper, there are 20 variables each of which is to be either included or not, leading to 220 different models. The coverage probability at any given value of the parameters provides an upper bound on the minimum coverage probability of the naive confidence interval. This paper derives a new Monte Carlo simulation estimator of the coverage probability, which uses conditioning for variance reduction. For these real‐life data, the gain in efficiency of this Monte Carlo simulation due to conditioning ranged from 2 to 6. The paper also presents a simple one‐dimensional search strategy for parameter values at which the coverage probability is relatively small. For these real‐life data, this search leads to parameter values for which the coverage probability of the naive 0.95 confidence interval is 0.79 for variable selection using the Akaike information criterion and 0.70 for variable selection using Bayes information criterion, showing that these confidence intervals are completely inadequate.

Confidence limits for estimates of totals from stratified samples, with application to medicare Part B overpayment audits

Superpopulation models are proposed that should be appropriate for modelling sample-based audits of Medicare payments and other overpayment situations. Simulations are used to estimate the coverage probabilities of confidence intervals formed using the standard Stratified Expansion and Combined Ratio estimators of the total. Despite severe departures from the usual model of normal deviations, these methods have actual coverage probabilities reasonably close to the nominal level specified by the US government's sampling guidelines. An exception occurs when all claims from a single sampling unit are either completely allowed, or completely denied, and for this situation an alternative is explored. A balanced sampling design is also examined, but shown to make no improvement over ordinary stratified samples used in conjunction with ratio estimates.

An Alternative to Cohen's Standardized Mean Difference Effect Size: A Robust Parameter and Confidence Interval in the Two Independent Groups Case.

The authors argue that a robust version of Cohen's effect size constructed by replacing population means with 20% trimmed means and the population standard deviation with the square root of a 20% Winsorized variance is a better measure of population separation than is Cohen's effect size. The authors investigated coverage probability for confidence intervals for the new effect size measure. The confidence intervals were constructed by using the noncentral t distribution and the percentile bootstrap. Over the range of distributions and effect sizes investigated in the study, coverage probability was better for the percentile bootstrap confidence interval.

Modeling the coverage probability of a confidence interval on the among-group variance component in the unbalanced random one-way model

The coverage probability of an approximate confidence interval on the among-group variance component, σ α 2 , in a one-way random model is modeled using generalized linear models techniques. The purpose of the proposed modeling is to derive an empirical relationship between the coverage probability on one hand, andk, n., o, and the model’s variance components on the other hand, wherek is the number of groups, n. is the total number of observations, and o is a measure of imbalance for the design used. The latter quantities serve as control variables in the derived model, and the coverage probability is treated as the response variable. Contour plots generated from this model can be easily used to depict the effects of the control variables on the coverage probability. In particular, the plots are utilized to compare four methods for constructing approximate confidence intervals on σ α 2 . Additional advantages of the derived model include prediction of the coverage probability for a given method using only specified values of the control variables, and the determination of operating conditions that result in improved coverage probability within the region of interest.

On Confidence Intervals for Genotype Relative Risks and Attributable Risks from Case Parent Trio Designs for Candidate-Gene Studies

Scherag et al. [Hum Hered 2002;54:210–217] recently proposed point estimates and asymptotic as well as exact confidence intervals for genotype relative risks (GRRs) and the attributable risk (AR) in case parent trio designs using single nucleotide polymorphism (SNP) data. The aim of this study was the investigation of coverage probabilities and bias in estimates if the marker locus is not identical to the disease locus. Using a variety of parameter constellations, including marker allele frequencies identical to and different from the SNP at the disease locus, we performed an analytical study to quantify the bias and a Monte-Carlo simulation study for quantifying both bias and coverage probabilities. No bias was observed if marker and trait locus coincided. Two parameters had a strong impact on coverage probabilities of confidence intervals and bias in point estimates if they did not coincide: the linkage disequilibrium (LD) parameter δ and the allele frequency at the marker SNP. If marker allele frequencies were different from the allele frequencies at the functional SNP, substantial biases occurred. Further, if δ between the marker and the disease locus was lower than the maximum possible δ, estimates were also biased. In general, biases were towards the null hypothesis for both GRRs and AR. If one GRR was not increased, as e.g. in a recessive genetic model, biases away from the null could be observed. If both GRRs were in identical directions and if both were substantially larger than 1, the bias always was towards the null. When applying point estimates and confidence intervals for GRRs and AR in candidate gene studies, great care is needed. Effect estimates are substantially biased towards the null if either the allele frequencies at the marker SNP and the true disease locus are different or if the LD between the marker SNP and the disease locus is not at its maximum. A bias away from the null occurs only in uncommon study situations; it is small and can therefore be ignored for applications.

Assessing Confidence Intervals for the Tail Index by Edgeworth Expansions for the Hill Estimator

We establish Edgeworth expansions for the distribution function of the centered and normalized Hill estimator for the reciprocal of the index of regular variation of the tail of a distribution function.The expansions are used to derive expansions for coverage probabilities of confidence intervals for the tail index based on the Hill estimator.

Use of Welch-Satterthwaite Approximation in Calibration of Voltage Standards

In certifying banks of solid state voltage standards, the customer is provided with predictive equations that are used to predict mean voltage and uncertainty in voltage. These equations are provided for both the bank average and the individual standards constituting the bank over the period of certification, which is typically a year. Use of the Welch-Satterthwaite Approximation (W-SA) can significantly reduce the number of calibrations required to provide satisfactory levels of uncertainty for the bank average. We first review the standard approach in metrology for expressing the uncertainty in a measurement system. We provide an intuitive discussion of the W-SA along with examples of its use and information on its usefulness when there are only a few degrees of freedom. A case study involving a bank of four solid state voltage standards is presented in detail. We also propose an analysis technique that can be used to investigate the usefulness and potential limitations of the W-SA when estimating measurement uncertainties. The analysis technique can be used to estimate approximation error, sampling error, and the coverage probabilities of confidence intervals computed using the W-SA.

An Improved Batch Means Procedure for Simulation Output Analysis

We formulate and evaluate the Automated Simulation Analysis Procedure (ASAP), an algorithm for steady-state simulation output analysis based on the method of nonover-lapping batch means (NOBM). ASAP delivers a confidence interval for an expected response that is centered on the sample mean of a portion of a simulation-generated time series and satisfies a user-specified absolute or relative precision requirement. ASAP operates as follows: The batch size is progressively increased until either (a) the batch means pass the von Neumann test for independence, and then ASAP delivers a classical NOBM confidence interval; or (b) the batch means pass the Shapiro-Wilk test for multivariate normality, and then ASAP delivers a correlation-adjusted confidence interval. The latter adjustment is based on an inverted Cornish-Fisher expansion for the classical NOBM t-ratio, where the terms of the expansion are estimated via an autoregressive-moving average time series model of the batch means. After determining the batch size and confidence-interval type, ASAP sequentially increases the number of batches until the precision requirement is satisfied. An extensive experimental study demonstrates the performance improvements achieved by ASAP versus well-known batch means procedures, especially in confidence-interval coverage probability.

A Note on the Efficiency of Sandwich Covariance Matrix Estimation

The sandwich estimator, also known as robust covariance matrix estimator, heteroscedasticity-consistent covariance matrix estimate, or empirical covariance matrix estimator, has achieved increasing use in the econometric literature as well as with the growing popularity of generalized estimating equations. Its virtue is that it provides consistent estimates of the covariance matrix for parameter estimates even when the fitted parametric model fails to hold or is not even specified. Surprisingly though, there has been little discussion of properties of the sandwich method other than consistency. We investigate the sandwich estimator in quasi-likelihood models asymptotically, and in the linear case analytically. We show that under certain circumstances when the quasi-likelihood model is correct, the sandwich estimate is often far more variable than the usual parametric variance estimate. The increased variance is a fixed feature of the method and the price that one pays to obtain consistency even when the parametric model fails or when there is heteroscedasticity. We show that the additional variability directly affects the coverage probability of confidence intervals constructed from sandwich variance estimates. In fact, the use of sandwich variance estimates combined with t-distribution quantiles gives confidence intervals with coverage probability falling below the nominal value. We propose an adjustment to compensate for this fact.

A REMARK ON SMALL-SAMPLE PROPERTIES OF LOGISTIC REGRESSION IN THREE-POINT DESIGNS

Nottingham and Birch [1] recently alleged that the maximum likelihood (ML) estimator of the steepness parameter in a logistic regression model could be seriously underestimated. They based their conclusion on a simulation study, investigating in particular a small-sample three-point design with a relatively large spacing between the doses. In the present work we study such situations in more detail and use complete enumeration to find the exact properties of the ML estimators. The result presented here show that the allegation by Nottingham and Birch was misleading. There is a substantial probability for an infinite outcome of , which appears to have been neglected by Nottingham and Birch. In fact, it will be demonstrated that the asymptotic normal approximation for fits quite well even with small samples, except in the upper tail where outcomes are infinite instead of large finite. The consequences for coverage probabilities of confidence intervals for both of the regression parameters are elucidated.