Valid Confidence Research Articles

Doubly-robust and heteroscedasticity-aware sample trimming for causal inference

SUMMARY A popular method for variance reduction in causal inference is propensity-based trimming, the practice of removing units with extreme propensities from the sample. This practice has theoretical grounding when the data are homoscedastic and the propensity model is parametric (Yang & Ding, 2018; Crump et al., 2009), but in modern settings where heteroscedastic data are analyzed with non-parametric models, existing theory fails to support current practice. In this work, we address this challenge by developing new methods and theory for sample trimming. Our contributions are three-fold: first, we describe novel procedures for selecting which units to trim. Our procedures differ from previous works in that we trim not only units with small propensities, but also units with extreme conditional variances. Second, we give new theoretical guarantees for inference after trimming. In particular, we show how to perform inference on the trimmed sub-population without requiring that our regressions converge at parametric rates. Instead, we make only fourth-root rate assumptions like those in the double machine learning literature. This result applies to conventional propensity-based trimming as well and thus may be of independent interest. Finally, we propose a bootstrap-based method for constructing simultaneously valid confidence intervals for multiple trimmed sub-populations, which are valuable for navigating the trade-off between sample size and variance reduction inherent in trimming. We validate our methods in simulation, on the 2007-2008 National Health and Nutrition Examination Survey, and on a semi-synthetic Medicare dataset and find promising results in all settings.

An effective and small sample-size valid confidence interval for isotonic dose-response curves by inverting a partial likelihood ratio test

A dose-response curve is essential for determining the safe dosage of a drug and is widely used in bioassay and in phase 1 clinical trials. It is generally accepted that the probability of death or the probability of dose-limiting toxicity is a non-decreasing function of the dose. This paper proposes and develops an effective point-wise confidence interval for an isotonic dose-response curve, a problem without a satisfactory solution so far. We show that one subset of the observations informs the lower limit while the other subset informs the upper limit. A partial likelihood ratio test using these subsets of observations is fully investigated. The resulting confidence interval is compared to three existing methods (Morris, Meyer, and Oron) in an extensive simulation study. The new method produces tight intervals while maintaining coverage. It is therefore preferred when maintaining an actual coverage probability is crucial as often in a regulatory environment. The new method extends to one-way ANOVA of a continuous response variable in a straightforward way. An R function is provided.

INFERENCE IN MILDLY EXPLOSIVE AUTOREGRESSIONS UNDER UNCONDITIONAL HETEROSKEDASTICITY

Mildly explosive autoregressions have been extensively employed in recent theoretical and applied econometric work to model the phenomenon of asset market bubbles. An important issue in this context concerns the construction of confidence intervals for the autoregressive parameter that represents the degree of explosiveness. Existing studies rely on intervals that are justified only under conditional homoskedasticity/heteroskedasticity. This paper studies the problem of constructing asymptotically valid confidence intervals in a mildly explosive autoregression where the innovations are allowed to be unconditionally heteroskedastic. The assumed variance process is general and can accommodate both deterministic and stochastic volatility specifications commonly adopted in the literature. Within this framework, we show that the standard heteroskedasticity- and autocorrelation-consistent estimate of the long-run variance converges in distribution to a nonstandard random variable that depends on nuisance parameters. Notwithstanding this result, the corresponding t-statistic is shown to still possess a standard normal limit distribution. To improve the quality of inference in small samples, we propose a dependent wild bootstrap-t procedure and establish its asymptotic validity under relatively weak conditions. Monte Carlo simulations demonstrate that our recommended approach performs favorably in finite samples relative to existing methods across a wide range of volatility specifications. Applications to international stock price indices and U.S. house prices illustrate the relevance of the advocated method in practice.

Test and Measure for Partial Mean Dependence Based on Machine Learning Methods

It is of importance to investigate the significance of a subset of covariates W for the response Y given covariates Z in regression modeling. To this end, we propose a significance test for the partial mean independence problem based on machine learning methods and data splitting. The test statistic converges to the standard Chi-squared distribution under the null hypothesis while it converges to a normal distribution under the fixed alternative hypothesis. Power enhancement and algorithm stability are also discussed. If the null hypothesis is rejected, we propose a partial Generalized Measure of Correlation (pGMC) to measure the partial mean dependence of Y given W after controlling for the nonlinear effect of Z. We present the appealing theoretical properties of the pGMC and establish the asymptotic normality of its estimator with the optimal root-N convergence rate. Furthermore, the valid confidence interval for the pGMC is also derived. As an important special case when there are no conditional covariates Z, we introduce a new test of overall significance of covariates for the response in a model-free setting. Numerical studies and real data analysis are also conducted to compare with existing approaches and to demonstrate the validity and flexibility of our proposed procedures. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments

Researchers are increasingly turning to machine learning (ML) algorithms to investigate causal heterogeneity in randomized experiments. Despite their promise, ML algorithms may fail to accurately ascertain heterogeneous treatment effects under practical settings with many covariates and small sample size. In addition, the quantification of estimation uncertainty remains a challenge. We develop a general approach to statistical inference for heterogeneous treatment effects discovered by a generic ML algorithm. We apply the Neyman’s repeated sampling framework to a common setting, in which researchers use an ML algorithm to estimate the conditional average treatment effect and then divide the sample into several groups based on the magnitude of the estimated effects. We show how to estimate the average treatment effect within each of these groups, and construct a valid confidence interval. In addition, we develop nonparametric tests of treatment effect homogeneity across groups, and rank-consistency of within-group average treatment effects. The validity of our methodology does not rely on the properties of ML algorithms because it is solely based on the randomization of treatment assignment and random sampling of units. Finally, we generalize our methodology to the cross-fitting procedure by accounting for the additional uncertainty induced by the random splitting of data.

Joint Statistical Inference for the Area under the ROC Curve and Youden Index under a Density Ratio Model

The receiver operating characteristic (ROC) curve is a valuable statistical tool in medical research. It assesses a biomarker’s ability to distinguish between diseased and healthy individuals. The area under the ROC curve (AUC) and the Youden index (J) are common summary indices used to evaluate a biomarker’s diagnostic accuracy. Simultaneously examining AUC and J offers a more comprehensive understanding of the ROC curve’s characteristics. In this paper, we utilize a semiparametric density ratio model to link the distributions of a biomarker for healthy and diseased individuals. Under this model, we establish the joint asymptotic normality of the maximum empirical likelihood estimator of (AUC,J) and construct an asymptotically valid confidence region for (AUC,J). Furthermore, we propose a new test to determine whether a biomarker simultaneously exceeds prespecified target values of AUC0 and J0 with the null hypothesis H0:AUC≤AUC0 or J≤J0 against the alternative hypothesis Ha:AUC>AUC0 and J>J0. Simulation studies and a real data example on Duchenne Muscular Dystrophy are used to demonstrate the effectiveness of our proposed method and highlight its advantages over existing methods.

Simultaneous Confidence Intervals for Partially Identified Parameters

This article extends the Imbens and Manski and Stoye confidence interval for a partially identified scalar parameter to a vector-valued parameter. The proposed method produces uniformly valid simultaneous confidence intervals for each dimension, or, equivalently, a rectangular confidence region that covers points in the identified set with a specified probability. The method applies when asymptotically normal estimates of upper and lower bounds for each dimension are available. The intervals are computationally simple and fast relative to methods based on test inversion or bootstrapped calibration, and do not suffer from the conservativity of projection-based approaches.

Sufficient Conditions for Central Limit Theorems and Confidence Intervals for Randomized Quasi-Monte Carlo Methods

Randomized quasi-Monte Carlo methods have been introduced with the main purpose of yielding a computable measure of error for quasi-Monte Carlo approximations through the implicit application of a central limit theorem over independent randomizations. But to increase precision for a given computational budget, the number of independent randomizations is usually set to a small value so that a large number of points are used from each randomized low-discrepancy sequence to benefit from the fast convergence rate of quasi-Monte Carlo. While a central limit theorem has been previously established for a specific but computationally expensive type of randomization, it is also known in general that fixing the number of randomizations and increasing the length of the sequence used for quasi-Monte Carlo can lead to a non-Gaussian limiting distribution. This paper presents sufficient conditions on the relative growth rates of the number of randomizations and the quasi-Monte Carlo sequence length to ensure a central limit theorem and also an asymptotically valid confidence interval. We obtain several results based on the Lindeberg condition for triangular arrays and expressed in terms of the regularity of the integrand and the convergence speed of the quasi-Monte Carlo method. We also analyze the resulting estimator’s convergence rate.

Abstract SY02-03: Stochastic modeling of clonal evolution in carcinogenesis

Abstract In order to develop strategies that can detect cancer earlier, we need to better understand the dynamics of cancer evolution during the human lifetime. Starting from gestation, this includes measuring both the timing of early somatic mutations that lead to clonal mosaicism as well as the growth rates of mutated clones that carry higher potential for malignant transformation. Continuous observation of this evolving clonal composition across a lifetime is not feasible, therefore mathematical models are needed to integrate data from longitudinal and cross-sectional samples and infer these dynamics. Stochastic models of clonal evolution offer such a framework and provide a means for predicting individual clone trajectories into the future. Furthermore, if measurements of evolution are to be utilized clinically for early detection and patient risk stratification, faster methods to apply these models to data are required. To this end, we derived methods using coalescent theory for single-cell DNA sequencing data that enable instantaneous estimation of the growth rate of a clone along with its confidence intervals. Our tool is available in an open source R package, cloneRate, and provides the first method for constructing valid confidence intervals of growth rates analytically without having to rely on Bayesian inference techniques and complex simulations. When applying our methods to recent datasets derived from normal and neoplastic human blood cells, we quantified increased fitness effects of multi-hit driver mutations and found that higher initial clone growth rates led to decreased time to cancer diagnosis in patients. We can also use these models to estimate other important biological parameters that are difficult to measure in vivo such as bounds for total number of hematopoietic stem cells and expansion rates in early development. Ultimately, the ability to easily and quickly parameterize clonal dynamics in individual patients will benefit future early detection efforts and screening strategies for many cancer types. Citation Format: Kit Curtius. Stochastic modeling of clonal evolution in carcinogenesis [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(7_Suppl):Abstract nr SY02-03.

On Sample Size Needed for Block Bootstrap Confidence Intervals to Have Desired Coverage Rates

Block bootstrap is widely used in constructing confidence intervals for parameters estimated from stationary time series. Theoretically, the method should provide valid confidence intervals as the length of the time series goes to infinity. In practice, however, it is necessary to know how large of a finite sample is required for block bootstrap confidence intervals to work well. This study aims to answer this question in a simple simulation setting where the data are generated from a first-order autoregressive process. The empirical coverage rates of several commonly used bootstrap confidence intervals for the mean, standard deviation, and the lag-1 autocorrelation coefficient are compared. A quite large sample is found necessary for the intervals to have the right coverage rates even when estimating a simple parameter like the mean. Some block bootstrap methods could fail when estimating the lag-1 autocorrelation. It is surprising that the coverage property even deteriorates as the sample size increases with some commonly used block bootstrap confidence intervals including the percentile intervals and bias-corrected intervals. KEYWORDS: Autocorrelation; Bias-Correction; Centering; Dependent Data; Percentile; Resampling; Simulation; Time Series

Analysis of Differentially Private Synthetic Data: A Measurement Error Approach

Differentially private (DP) synthetic datasets have been receiving significant attention from academia, industry, and government. However, little is known about how to perform statistical inference using DP synthetic datasets. Naive approaches that do not take into account the induced uncertainty due to the DP mechanism will result in biased estimators and invalid inferences. In this paper, we present a class of maximum likelihood estimator (MLE)-based easy-to-implement bias-corrected DP estimators with valid asymptotic confidence intervals (CI) for parameters in regression settings, by establishing the connection between additive DP mechanisms and measurement error models. Our simulation shows that our estimator has comparable performance to the widely used sufficient statistic perturbation (SSP) algorithm in some scenarios but with the advantage of releasing a synthetic dataset and obtaining statistically valid asymptotic CIs, which can achieve better coverage when compared to the naive CIs obtained by ignoring the DP mechanism.

Resampling-based confidence intervals and bands for the average treatment effect in observational studies with competing risks

The g-formula can be used to estimate the treatment effect while accounting for confounding bias in observational studies. With regard to time-to-event endpoints, possibly subject to competing risks, the construction of valid pointwise confidence intervals and time-simultaneous confidence bands for the causal risk difference is complicated, however. A convenient solution is to approximate the asymptotic distribution of the corresponding stochastic process by means of resampling approaches. In this paper, we consider three different resampling methods, namely the classical nonparametric bootstrap, the influence function equipped with a resampling approach as well as a martingale-based bootstrap version, the so-called wild bootstrap. For the latter, three sub-versions based on differing distributions of the underlying random multipliers are examined. We set up a simulation study to compare the accuracy of the different techniques, which reveals that the wild bootstrap should in general be preferred if the sample size is moderate and sufficient data on the event of interest have been accrued. For illustration, the resampling methods are further applied to data on the long-term survival in patients with early-stage Hodgkin’s disease.

Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods

This article investigates the problem of online statistical inference of model parameters in stochastic optimization problems via the Kiefer-Wolfowitz algorithm with random search directions. We first present the asymptotic distribution for the Polyak-Ruppert-averaging type Kiefer-Wolfowitz (AKW) estimators, whose asymptotic covariance matrices depend on the distribution of search directions and the function-value query complexity. The distributional result reflects the tradeoff between statistical efficiency and function query complexity. We further analyze the choice of random search directions to minimize certain summary statistics of the asymptotic covariance matrix. Based on the asymptotic distribution, we conduct online statistical inference by providing two construction procedures of valid confidence intervals. Supplementary materials for this article are available online.

An identification and testing strategy for proxy-SVARs with weak proxies

When proxies (external instruments) used to identify target structural shocks are weak, inference in proxy-SVARs (SVAR-IVs) is nonstandard and the construction of asymptotically valid confidence sets for the impulse responses of interest requires weak-instrument robust methods. In the presence of multiple target shocks, test inversion techniques require extra restrictions on the proxy-SVAR parameters other than those implied by the proxies that may be difficult to interpret and test. We show that frequentist asymptotic inference in these situations can be conducted through Minimum Distance estimation and standard asymptotic methods if the proxy-SVAR can be identified by using ‘strong’ instruments for the non-target shocks; i.e., the shocks which are not of primary interest in the analysis. The suggested identification strategy hinges on a novel pre-test for the null of instrument relevance, based on bootstrap resampling, which is not subject to pre-testing issues. Specifically, the validity of post-test asymptotic inferences remains unaffected by the test outcomes due to an asymptotic independence result between the bootstrap and non-bootstrap statistics. The test is robust to conditionally heteroskedastic and/or zero-censored proxies, is computationally straightforward and applicable regardless of the number of shocks being instrumented. Some illustrative examples show the empirical usefulness of the suggested identification and testing strategy.

Asymptotics of K-Fold Cross Validation

This paper investigates the asymptotic distribution of the K-fold cross validation error in an i.i.d. setting. As the number of observations n goes to infinity while keeping the number of folds K fixed, the K-fold cross validation error is √ n-consistent for the expected out-of-sample error and has an asymptotically normal distribution. A consistent estimate of the asymptotic variance is derived and used to construct asymptotically valid confidence intervals for the expected out-of-sample error. A hypothesis test is developed for comparing two estimators’ expected out-of-sample errors and a subsampling procedure is used to obtain critical values. Monte Carlo simulations demonstrate the asymptotic validity of our confidence intervals for the expected out-of-sample error and investigate the size and power properties of our test. In our empirical application, we use our estimator selection test to compare the out-of-sample predictive performance of OLS, Neural Networks, and Random Forests for predicting the sale price of a domain name in a GoDaddy expiry auction.

Prediction-powered inference.

Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients without making any assumptions about the machine-learning algorithm that supplies the predictions. Furthermore, more accurate predictions translate to smaller confidence intervals. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning. The benefits of prediction-powered inference were demonstrated with datasets from proteomics, astronomy, genomics, remote sensing, census analysis, and ecology.

Bump hunting through density curvature features

Bump hunting deals with finding in sample spaces meaningful data subsets known as bumps. These have traditionally been conceived as modal or concave regions in the graph of the underlying density function. We define an abstract bump construct based on curvature functionals of the probability density. Then, we explore several alternative characterizations involving derivatives up to second order. In particular, a suitable implementation of Good and Gaskins’ original concave bumps is proposed in the multivariate case. Moreover, we bring to exploratory data analysis concepts like the mean curvature and the Laplacian that have produced good results in applied domains. Our methodology addresses the approximation of the curvature functional with a plug-in kernel density estimator. We provide theoretical results that assure the asymptotic consistency of bump boundaries in the Hausdorff distance with affordable convergence rates. We also present asymptotically valid and consistent confidence regions bounding curvature bumps. The theory is illustrated through several use cases in sports analytics with datasets from the NBA, MLB and NFL. We conclude that the different curvature instances effectively combine to generate insightful visualizations.

Estimation of SARS-CoV-2 Seroprevalence in Central North Carolina: Accounting for Outcome Misclassification in Complex Sample Designs.

Population-based seroprevalence studies are crucial to understand community transmission of COVID-19 and guide responses to the pandemic. Seroprevalence is typically measured from diagnostic tests with imperfect sensitivity and specificity. Failing to account for measurement error can lead to biased estimates of seroprevalence. Methods to adjust seroprevalence estimates for the sensitivity and specificity of the diagnostic test have largely focused on estimation in the context of convenience sampling. Many existing methods are inappropriate when data are collected using a complex sample design. We present methods for seroprevalence point estimation and confidence interval construction that account for imperfect test performance for use with complex sample data. We apply these methods to data from the Chatham County COVID-19 Cohort (C4), a longitudinal seroprevalence study conducted in central North Carolina. Using simulations, we evaluate bias and confidence interval coverage for the proposed estimator compared with a standard estimator under a stratified, three-stage cluster sample design. We obtained estimates of seroprevalence and corresponding confidence intervals for the C4 study. SARS-CoV-2 seroprevalence increased rapidly from 10.4% in January to 95.6% in July 2021 in Chatham County, North Carolina. In simulation, the proposed estimator demonstrates desirable confidence interval coverage and minimal bias under a wide range of scenarios. We propose a straightforward method for producing valid estimates and confidence intervals when data are based on a complex sample design. The method can be applied to estimate the prevalence of other infections when estimates of test sensitivity and specificity are available.

Martingale Methods for Sequential Estimation of Convex Functionals and Divergences

We present a unified technique for sequential estimation of convex divergences between distributions, including integral probability metrics like the kernel maximum mean discrepancy, $\varphi$-divergences like the Kullback-Leibler divergence, and optimal transport costs, such as powers of Wasserstein distances. This is achieved by observing that empirical convex divergences are (partially ordered) reverse submartingales with respect to the exchangeable filtration, coupled with maximal inequalities for such processes. These techniques appear to be complementary and powerful additions to the existing literature on both confidence sequences and convex divergences. We construct an offline-to-sequential device that converts a wide array of existing offline concentration inequalities into time-uniform confidence sequences that can be continuously monitored, providing valid tests or confidence intervals at arbitrary stopping times. The resulting sequential bounds pay only an iterated logarithmic price over the corresponding fixed-time bounds, retaining the same dependence on problem parameters (like dimension or alphabet size if applicable). These results are also applicable to more general convex functionals -- like the negative differential entropy, suprema of empirical processes, and V-Statistics -- and to more general processes satisfying a key leave-one-out property.

Inference on the best policies with many covariates

Understanding the impact of the most effective policies or treatments on a response variable of interest is desirable in many empirical works in economics, statistics and other disciplines. Due to the widespread winner’s curse phenomenon, conventional statistical inference assuming that the top policies are chosen independent of the random sample may lead to overly optimistic evaluations of the best policies. In recent years, given the increased availability of large datasets, such an issue can be further complicated when researchers include many covariates to estimate the policy or treatment effects in an attempt to control for potential confounders. In this manuscript, to simultaneously address the above-mentioned issues, we propose a resampling-based procedure that not only lifts the winner’s curse in evaluating the best policies observed in a random sample, but also is robust to the presence of many covariates. The proposed inference procedure yields accurate point estimates and valid frequentist confidence intervals that achieve the exact nominal level as the sample size goes to infinity for multiple best policy effect sizes. We illustrate the finite-sample performance of our approach through Monte Carlo experiments and two empirical studies, evaluating the most effective policies in charitable giving and the most beneficial group of workers in the National Supported Work program.