Correct Coverage Research Articles

A Primer of Quasi-Exact Tests for Randomized Clinical Trials with Binary Endpoints

For planned clinical trials with a binary endpoint, standard statistical tests can be very inaccurate in their error control, especially when the trial is unbalanced. Specifically, Type 1 error can greatly exceed a nominal target like 5%, even for quite large sample sizes, and power can be much less than a nominal target such as 80%. Confidence limits may not have the correct coverage. The purpose of this article is first to give clinical trialists an appreciation of this fact. We then describe well-studied alternatives to the standard methods, which are simple in conception but can be quite challenging to implement. These contemporary methods in frequentist statistical theory, known as quasi-exact methods, have been neglected in medical applications. We provide an R-package that can be used to provide accurate inference results based on our recommended quasi-exact test. The package will also calculate the exact size and power of the test, once the control and treatment samples sizes are finalized. All proposals are illustrated with a real-world example on the treatment of prostate cancer. We would recommend practitioners to use quasi-exact methods in the analysis of randomized clinical trials.

On uniform confidence intervals for the tail index and the extreme quantile

This paper presents two results concerning uniform confidence intervals for the tail index and the extreme quantile. First, we show that there exists a lower bound of the length for confidence intervals that satisfy the correct uniform coverage over a nonparametric family of tail distributions. Second, in light of the impossibility result, we construct honest confidence intervals that are uniformly valid by incorporating the worst-case bias in the nonparametric family. The proposed method is applied to simulated data and real data of financial time series.

Model-Based Prediction for Small Domains Using Covariates: A Comparison of Four Methods

Abstract We consider methods for model-based small area estimation when the number of areas with sampled data is a small fraction of the total areas for which estimates are required. Abundant auxiliary information is available from the survey for all the sampled areas. Further, through an external source, there is information for all areas. The goal is to use auxiliary variables to predict the outcome of interest for all areas. We compare areal-level random forests and LASSO approaches to a frequentist forward variable selection approach and a Bayesian shrinkage method using a horseshoe prior. Further, to measure the uncertainty of estimates obtained from random forests and the LASSO, we propose a modification of the split conformal procedure that relaxes the assumption of exchangeable data. We show that the proposed method yields intervals with the correct coverage rate and this is confirmed through a simulation study. This work is motivated by Ghanaian data available from the sixth Ghana Living Standards Survey (GLSS) and the 2010 Population and Housing Census, in the Greater Accra Metropolitan Area (GAMA) region, which comprises eight districts that are further divided into enumeration areas (EAs). We estimate the areal mean household log consumption using both datasets. The outcome variable is measured only in the GLSS for 3 percent of all the EAs (136 out of 5019) and 174 potential covariates are available in both datasets. In the application, among the four modeling methods considered, the Bayesian shrinkage performed the best in terms of bias, mean squared error (MSE), and prediction interval coverages and scores, as assessed through a cross-validation study. We find substantial between-area variation with the estimated log consumption showing a 1.3-fold variation across the GAMA region. The western areas are the poorest while the Accra Metropolitan Area district has the richest areas.

Anatomy, Etiology, Management, and Medico-Legal Implications of Botulinum-induced Blepharoptosis

Abstract: Botulinum toxin injections, a popular aesthetic treatment, have over 7.4 million beneficiaries in the U.S. Despite their safety record, these injections pose potential complications. It is essential for aesthetic practitioners to manage these complications with the least impact on the patient. Upper eyelid ptosis, though rare, is a significant side effect of botulinum toxin injections. Through our study, we have identified the etiology, anatomy, and therapeutic management of botulinum- induced blepharoptosis. Hence, the goal of this study was to identify the basic aetiology of blepharoptosis and manage this complication, as well as discuss the basis of medico-legal implications involving this popular drug. The complex medico-legal implications of botulinum toxin-induced blepharoptosis call for continuous discourse, education, and clarity on drug-use legal standards. With evolving global and Italian legislation, practitioners must ensure they meet care standards, weighing treatment benefits against potential legal and ethical outcomes. Blepharoptosis is a rare but significant complication of botulinum-type injections. Etiology and thorough anatomy are crucial for avoiding this complication and handling it with the least impact on the patient. Medico-legal implications are currently not fully established, but the basis of aesthetic treatment standards, as well as continuing medical education, will ensure correct medico-legal coverage of such complications.

On exact randomization-based covariate-adjusted confidence intervals.

Randomization-based inference using the Fisher randomization test allows for the computation of Fisher-exact P-values, making it an attractive option for the analysis of small, randomized experiments with non-normal outcomes. Two common test statistics used to perform Fisher randomization tests are the difference-in-means between the treatment and control groups and the covariate-adjusted version of the difference-in-means using analysis of covariance. Modern computing allows for fast computation of the Fisher-exact P-value, but confidence intervals have typically been obtained by inverting the Fisher randomization test over a range of possible effect sizes. The test inversion procedure is computationally expensive, limiting the usage of randomization-based inference in applied work. A recent paper by Zhu and Liu developed a closed form expression for the randomization-based confidence interval using the difference-in-means statistic. We develop an important extension of Zhu and Liu to obtain a closed form expression for the randomization-based covariate-adjusted confidence interval and give practitioners a sufficiency condition that can be checked using observed data and that guarantees that these confidence intervals have correct coverage. Simulations show that our procedure generates randomization-based covariate-adjusted confidence intervals that are robust to non-normality and that can be calculated in nearly the same time as it takes to calculate the Fisher-exact P-value, thus removing the computational barrier to performing randomization-based inference when adjusting for covariates. We also demonstrate our method on a re-analysis of phase I clinical trial data.

How to evaluate uncertainty estimates in machine learning for regression?

As neural networks become more popular, the need for accompanying uncertainty estimates increases. There are currently two main approaches to test the quality of these estimates. Most methods output a density. They can be compared by evaluating their loglikelihood on a test set. Other methods output a prediction interval directly. These methods are often tested by examining the fraction of test points that fall inside the corresponding prediction intervals. Intuitively, both approaches seem logical. However, we demonstrate through both theoretical arguments and simulations that both ways of evaluating the quality of uncertainty estimates have serious flaws. Firstly, both approaches cannot disentangle the separate components that jointly create the predictive uncertainty, making it difficult to evaluate the quality of the estimates of these components. Specifically, the quality of a confidence interval cannot reliably be tested by estimating the performance of a prediction interval. Secondly, the loglikelihood does not allow a comparison between methods that output a prediction interval directly and methods that output a density. A better loglikelihood also does not necessarily guarantee better prediction intervals, which is what the methods are often used for in practice. Moreover, the current approach to test prediction intervals directly has additional flaws. We show why testing a prediction or confidence interval on a single test set is fundamentally flawed. At best, marginal coverage is measured, implicitly averaging out overconfident and underconfident predictions. A much more desirable property is pointwise coverage, requiring the correct coverage for each prediction. We demonstrate through practical examples that these effects can result in favouring a method, based on the predictive uncertainty, that has undesirable behaviour of the confidence or prediction intervals. Finally, we propose a simulation-based testing approach that addresses these problems while still allowing easy comparison between different methods. This approach can be used for the development of new uncertainty quantification methods.

Confidence bands for survival curves from outcome‐dependent stratified samples

AbstractWe consider the construction of confidence bands for survival curves under the outcome‐dependent stratified sampling. A main challenge of this design is that data are a biased dependent sample due to stratification and sampling without replacement. Most literature on regression approximates this design by Bernoulli sampling but variance is generally overestimated. Even with this approximation, the limiting distribution of the inverse probability weighted Kaplan–Meier estimator involves a general Gaussian process, and hence quantiles of its supremum is not analytically available. In this paper, we provide a rigorous asymptotic theory for the weighted Kaplan–Meier estimator accounting for dependence in the sample. We propose the novel hybrid method to both simulate and bootstrap parts of the limiting process to compute confidence bands with asymptotically correct coverage probability. Simulation study indicates that the proposed bands are appropriate for practical use. A Wilms tumor example is presented.

Locally robust inference for non‐Gaussian SVAR models

All parameters in structural vector autoregressive (SVAR) models are locally identified when the structural shocks are independent and follow non‐Gaussian distributions. Unfortunately, standard inference methods that exploit such features of the data for identification fail to yield correct coverage for structural functions of the model parameters when deviations from Gaussianity are small. To this extent, we propose a locally robust semiparametric approach to conduct hypothesis tests and construct confidence sets for structural functions in SVAR models. The methodology fully exploits non‐Gaussianity when it is present, but yields correct size/coverage for local‐to‐Gaussian densities. Empirically, we revisit two macroeconomic SVAR studies where we document mixed results. For the oil price model of Kilian and Murphy (2012), we find that non‐Gaussianity can robustly identify reasonable confidence sets, whereas for the labor supply–demand model of Baumeister and Hamilton (2015) this is not the case. Moreover, these exercises highlight the importance of using weak identification robust methods to assess estimation uncertainty when using non‐Gaussianity for identification.

Frequentist model averaging for analysis of dose-response in epidemiologic studies with complex exposure uncertainty.

In epidemiologic studies, association estimates of an exposure with disease outcomes are often biased when the uncertainties of exposure are ignored. Consequently, corresponding confidence intervals (CIs) will not have correct coverage. This issue is particularly problematic when exposures must be reconstructed from physical measurements, for example, for environmental or occupational radiation doses that were received by a study population for which radiation doses cannot be measured directly. To incorporate complex uncertainties in reconstructed exposures, the two-dimensional Monte Carlo (2DMC) dose estimation method has been proposed and used in various dose reconstruction efforts. The 2DMC method generates multiple exposure realizations from dosimetry models that incorporate various sources of errors to reflect the uncertainty of the dose distribution as well as the uncertainties in individual doses in the exposed population. Traditional measurement-error model approaches, typically based on using mean doses in the dose-exposure analysis, do not fully account exposure uncertainties. A recently developed statistical approach that overcomes many of these limitations by analyzing multiple exposure realizations in relation to disease risk is Bayesian model averaging (BMA). The analytic advantage of the BMA is its ability to better accommodate complex exposure uncertainty in the risk estimation, but a practical. Drawback is its significant computational complexity. In this present paper, we propose a novel frequentist model averaging (FMA) approach which has all the analytical advantages of the BMA method but is much simpler to implement and computationally faster. We show in simulations that, like BMA, FMA yields 95% confidence intervals for association parameters that close to 95% coverage rate. In simulations, the FMA has shorter length of CIs than those of another frequentist approach, the corrected information matrix (CIM) method. We illustrate the similarities in performance of BMA and FMA from a study of exposures from radioactive fallout in Kazakhstan.

Enrollment forecast for clinical trials at the portfolio planning phase based on site-level historical data.

An accurate forecast of a clinical trial enrollment timeline at the planning phase is of great importance to both corporate strategic planning and trial operational excellence. The naive approach often calculates an average enrollment rate from historical data and generates an inaccurate prediction based on a linear trend with the average rate. Under the traditional framework of a Poisson-Gamma model, site activation delays are often modeled with either fixed initiation time or a simple random distribution while incorporating the user-provided site planning information to achieve good forecast accuracy. However, such user-provided information is not available at the early portfolio planning stage. We present a novel statistical approach based on generalized linear mixed-effects models and the use of non-homogeneous Poisson processes through the Bayesian framework to model the country initiation, site activation, and subject enrollment sequentially in a systematic fashion. We validate the performance of our proposed enrollment modeling framework based on a set of 25 preselected studies from four therapeutic areas. Our modeling framework shows a substantial improvement in prediction accuracy in comparison to the traditional statistical approach. Furthermore, we show that our modeling and simulation approach calibrates the data variability appropriately and gives correct coverage rates for prediction intervals of various nominal levels. Finally, we demonstrate the use of our approach to generate the predicted enrollment curves through time with confidence bands overlaid.

An eigenvector-assisted estimation framework for signal-plus-noise matrix models

Summary In this paper, we develop an eigenvector-assisted estimation framework for a collection of signal-plus-noise matrix models arising in high-dimensional statistics and many applications. The framework is built upon a novel asymptotically unbiased estimating equation using the leading eigenvectors of the data matrix. However, the estimator obtained by directly solving the estimating equation could be numerically unstable in practice and lacks robustness against model misspecification. We propose to use the quasi-posterior distribution by exponentiating a criterion function whose maximizer coincides with the estimating equation estimator. The proposed framework can incorporate heteroskedastic variance information, but does not require the complete specification of the sampling distribution and is also robust to the potential misspecification of the distribution of the noise matrix. Computationally, the quasi-posterior distribution can be obtained via a Markov chain Monte Carlo sampler, which exhibits superior numerical stability over some of the existing optimization-based estimators and is straightforward for uncertainty quantification. Under mild regularity conditions, we establish the large sample properties of the quasi-posterior distributions. In particular, the quasi-posterior credible sets have the correct frequentist nominal coverage probability provided that the criterion function is carefully selected. The validity and usefulness of the proposed framework are demonstrated through the analysis of synthetic datasets and the real-world ENZYMES network datasets.

Robust variance estimation in small meta-analysis with the standardized mean difference.

Conventional random-effects models in meta-analysis rely on large sample approximations instead of exact small sample results. While random-effects methods produce efficient estimates and confidence intervals for the summary effect have correct coverage when the number of studies is sufficiently large, we demonstrate that conventional methods result in confidence intervals that are not wide enough when the number of studies is small, depending on the configuration of sample sizes across studies, the degree of true heterogeneity and number of studies. We introduce two alternative variance estimators with better small sample properties, investigate degrees of freedom adjustments for computing confidence intervals, and study their effectiveness via simulation studies.

Deep Bayesian survival analysis of rail useful lifetime

Reliable estimation of rail useful lifetime can provide valuable information for predictive maintenance in railway systems. However, in most cases, lifetime data is incomplete because not all pieces of rail experience failure by the end of the study horizon, a problem known as censoring. Ignoring or otherwise mistreating the censored cases might lead to false conclusions. Survival approach is particularly designed to handle censored data for analysing the expected duration of time until one event occurs, which is rail failure in this paper. This paper proposes a deep Bayesian survival approach named BNN-Surv to properly handle censored data for rail useful lifetime modelling. The proposed BNN-Surv model applies the deep neural network in the survival approach to capture the non-linear relationship between covariates and rail useful lifetime. To consider and quantify uncertainty in the model, Monte Carlo dropout, regarded as the approximate Bayesian inference, is incorporated into the deep neural network to provide the confidence interval of the estimated lifetime. The proposed approach is implemented on a four-year dataset including track geometry monitoring data, track characteristics data, various types of defect data, and maintenance and replacement (M&R) data collected from a section of railway tracks in Australia. Through extensive evaluation, including Concordance index (C-index) and root mean square error (RMSE) for evaluating model performance, as well as a proposed CW-index for evaluating uncertainty estimations, the effectiveness of the proposed approach is confirmed. The results show that, compared with other commonly used models, the proposed approach can achieve the best concordance index (C-index) of 0.80, and the estimated rail useful lifetimes are closer to real lifetimes. In addition, the proposed approach can provide the confidence interval of the estimated lifetime, with a correct coverage of 81% of the actual lifetime when the confidence interval is 1.38, which is more useful than point estimates in decision-making and maintenance planning of railroad systems.

Clustering of trajectories with mixed effects classification model: Inference taking into account classification uncertainties.

Classifying patient biomarker trajectories into groups has become frequent in clinical research. Mixed effects classification models can be used to model the heterogeneity of longitudinal data. The estimated parameters of typical trajectories and the partition can be provided by the classification version of the expectation maximization algorithm, named CEM. However, the variance of the parameter estimates obtained underestimates the true variance because classification uncertainties are not taken into account. This article takes into account these uncertainties by using the stochastic EM algorithm (SEM), a stochastic version of the CEM algorithm, after convergence of the CEM algorithm. The simulations showed correct coverage probabilities of the 95% confidence intervals (close to 95% except for scenarios with high bias in typical trajectories). The method was applied on a trial, called low-cyclo, that compared the effects of low vs standard cyclosporine A doses on creatinine levels after cardiac transplantation. It identified groups of patients for whom low-dose cyclosporine may be relevant, but with high uncertainty on the dose-effect estimate.

Cross-Validation: What Does It Estimate and How Well Does It Do It?

Cross-validation is a widely used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow’s C p . Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail. Lastly, our analysis also shows that when producing confidence intervals for prediction accuracy with simple data splitting, one should not refit the model on the combined data, since this invalidates the confidence intervals. Supplementary materials for this article are available online.

Cross‐sectional Gravity Models, PPML Estimation, and the Bias Correction of the Two‐Way Cluster‐Robust Standard Errors*

AbstractIn cross‐section gravity models the two‐way cluster‐robust standard errors of the Poisson pseudo maximum likelihood (PPML) estimates tend to be considerably downward biased. However, two‐way clustering can be avoided if intra‐cluster correlation is induced by country‐specific trade shocks with uniform pass through (equi‐correlation) and the gravity model includes exporter and importer country fixed effects. In this case the pseudo‐within‐transformation of the PPML estimator projects out the corresponding components of the disturbances. In Monte Carlo simulations the Pustejovsky and Tipton (2018) bias correction for independent disturbances (i.e. ignoring clustering) reveals just a small downward bias of the estimated standard errors and confidence intervals with nearly correct coverage rates. Under deviations from equi‐correlation the bias is somewhat larger, but still comparable to the bias of the cluster‐robust standard errors with Pustejovsky and Tipton (2018) bias correction.

Hybrid confidence intervals for informative uniform asymptotic inference after model selection

Abstract I propose a new type of confidence interval for correct asymptotic inference after using data to select a model of interest without assuming any model is correctly specified. This hybrid confidence interval is constructed by combining techniques from the selective inference and post-selection inference literatures to yield a short confidence interval across a wide range of data realizations. I show that hybrid confidence intervals have correct asymptotic coverage, uniformly over a large class of probability distributions that do not bound scaled model parameters. I illustrate the use of these confidence intervals in the problem of inference after using the lasso objective function to select a regression model of interest and provide evidence of their desirable length and coverage properties in small samples via a set of Monte Carlo experiments that entail a variety of different data distributions as well as an empirical application to the predictors of diabetes disease progression.

A practical revealed preference model for separating preferences and availability effects in marriage formation.

Many demographic problems require models for partnership formation. We consider a model for matchings within a bipartite population where individuals have utility for people based on observed and unobserved characteristics. It represents both the availability of potential partners of different types and the preferences of individuals for such people. We develop an estimator for the preference parameters based on sample survey data on partnerships and population composition. We conduct simulation studies based on the Survey of Income and Program Participation showing that the estimator recovers preference parameters that are invariant under different population availabilities and has the correct confidence coverage.

Simultaneous confidence intervals for quantile differences of several heterogeneous normal populations: With application to vitamin D supplement treatment on colorectal cancer patients.

In many applications, comparing the q-quantiles of several normal populations are more advantageous than comparing their means. In this paper, we consider the problem of constructing simultaneous confidence intervals (SCIs) for quantile differences of several heterogeneous normal distributions. To the best of our knowledge, this problem remains unsolved. We propose a novel method for constructing a set of SCI. We propose two new sets of SCI by using the proposed technique and discuss two classic and two simulation-based SCIs. We show that the proposed classic SCIs are conservative for all population parameters configuration. We also show that the simulation-based SCIs have correct coverage probability asymptotically. We then compare these six sets of SCI in terms of average volume and coverage probability via an extensive simulation study. Results show that one of the proposed classic SCI can be recommended. Finally, the proposed methods are illustrated by a real example that is a vitamin D study on colorectal cancer patients.

Cover your bases: asymptotic distributions of the profile likelihood ratio when constraining effective field theories in high-energy physics

We investigate the asymptotic distribution of the profile likelihood ratio (PLR) when constraining effective field theories (EFTs) and show that Wilks’ theorem is often violated, meaning that we should not assume the PLR to follow a \chi^2χ2-distribution. We derive the correct asymptotic distributions when either one or two real EFT couplings modulate observable cross sections with a purely linear or quadratic dependence. We then discover that when both the linear and quadratic terms contribute, the PLR distribution does not have a simple form. In this case we provide a partly-numerical solution for the one-parameter case. Using a novel approach, we find that the constants which define our asymptotic distributions may be obtained experimentally using a profile of the Asimov likelihood contour. Our results may be immediately used to obtain the correct coverage when deriving real-world EFT constraints using the PLR as a test-statistic.