Valid Confidence Research Articles

Confidence-based Estimators for Predictive Performance in Model Monitoring

After a machine learning model has been deployed into production, its predictive performance needs to be monitored. Ideally, such monitoring can be carried out by comparing the model’s predictions against ground truth labels. For this to be possible, the ground truth labels must be available relatively soon after inference. However, there are many use cases where ground truth labels are available only after a significant delay, or in the worst case, not at all. In such cases, directly monitoring the model’s predictive performance is impossible. Recently, novel methods for estimating the predictive performance of a model when ground truth is unavailable have been developed. Many of these methods leverage model confidence or other uncertainty estimates and are experimentally compared against a naive baseline method, namely Average Confidence (AC), which estimates model accuracy as the average of confidence scores for a given set of predictions. However, until now the theoretical properties of the AC method have not been properly explored. In this paper, we bridge this gap by reviewing the AC method and show that under certain general assumptions, it is an unbiased and consistent estimator of model accuracy. We also augment the AC method by deriving valid confidence intervals for the estimates it produces. These contributions elevate AC from an ad-hoc estimator to a principled one, encouraging its use in practice. We complement our theoretical results with empirical experiments, comparing AC against more complex estimators in a monitoring setting under covariate shift. We conduct our experiments using synthetic datasets, which allow for full control over the nature of the shift. Our experiments with binary classifiers show that the AC method is able to beat other estimators in many cases. However, the comparative quality of the different estimators is found to be heavily case-dependent.

Ordered correlation forest

. Empirical studies in various social sciences often involve categorical outcomes with inherent ordering, such as self-evaluations of subjective well-being and self-assessments in health domains. While ordered choice models, such as the ordered logit and ordered probit, are popular tools for analyzing these outcomes, they may impose restrictive parametric and distributional assumptions. This article introduces a novel estimator, the ordered correlation forest, that can naturally handle non linearities in the data and does not assume a specific error term distribution. The proposed estimator modifies a standard random forest splitting criterion to build a collection of forests, each estimating the conditional probability of a single class. Under an “honesty” condition, predictions are consistent and asymptotically normal. The weights induced by each forest are used to obtain standard errors for the predicted probabilities and the covariates’ marginal effects. Evidence from synthetic data shows that the proposed estimator features a superior prediction performance than alternative forest-based estimators and demonstrates its ability to construct valid confidence intervals for the covariates’ marginal effects. Comparisons using various real-world data sets further highlight the advantages of forest-based estimators over parametric models in larger samples while showing that the ordered correlation forest remains competitive in smaller samples.

Multiple Structural Breaks in Interactive Effects Panel Data Models

ABSTRACTThis paper develops new econometric methods for multiple structural break detection in panel data models with interactive fixed effects. The new methods include tests for the presence of structural breaks, estimators for the number of breaks and their location, and a method for constructing asymptotically valid break date confidence intervals. The new methodology is applied to a large panel of US banks for a period characterized by massive quantitative easing programs aimed at lessening the impact of the global financial crisis and the COVID‐19 pandemic. The question we ask is as follows: Have these programs been successful in spurring bank lending in the US economy? The short answer turns out to be: “No”.

-Penalized Multinomial Regression: Estimation, Inference, and Prediction, With an Application to Risk Factor Identification for Different Dementia Subtypes.

High-dimensional multinomial regression models are very useful in practice but have received less research attention than logistic regression models, especially from the perspective of statistical inference. In this work, we analyze the estimation and prediction error of the contrast-based -penalized multinomial regression model and extend the debiasing method to the multinomial case, providing a valid confidence interval for each coefficient and value of the individual hypothesis test. We also examine cases of model misspecification and non-identically distributed data to demonstrate the robustness of our method when some assumptions are violated. We apply the debiasing method to identify important predictors in the progression into dementia of different subtypes. Results from extensive simulations show the superiority of the debiasing method compared to other inference methods.

An Effective and Small Sample-Size Valid Confidence Interval for Isotonic Dose–Response Curves by Inverting a Partial Likelihood Ratio Test

A dose–response curve is essential for determining the safe dosage of a drug and is widely used in bioassay and in phase 1 clinical trials. It is generally accepted that the probability of death or the probability of dose-limiting toxicity is a nondecreasing function of the dose. This article proposes and develops an effective point-wise confidence interval for an isotonic dose–response curve, a problem without a satisfactory solution so far. We show that one subset of the observations informs the lower limit while the other subset informs the upper limit. A partial likelihood ratio test using these subsets of observations is fully investigated. The resulting confidence interval is compared to three existing methods (Morris, Meyer, and Oron) in an extensive simulation study. The new method produces tight intervals while maintaining coverage. It is therefore preferred when maintaining an actual coverage probability is crucial as often in a regulatory environment. The new method extends to one-way ANOVA of a continuous response variable in a straightforward way. An R function is provided.

Doubly-robust and heteroscedasticity-aware sample trimming for causal inference

SUMMARY A popular method for variance reduction in causal inference is propensity-based trimming, the practice of removing units with extreme propensities from the sample. This practice has theoretical grounding when the data are homoscedastic and the propensity model is parametric (Yang & Ding, 2018; Crump et al., 2009), but in modern settings where heteroscedastic data are analyzed with non-parametric models, existing theory fails to support current practice. In this work, we address this challenge by developing new methods and theory for sample trimming. Our contributions are three-fold: first, we describe novel procedures for selecting which units to trim. Our procedures differ from previous works in that we trim not only units with small propensities, but also units with extreme conditional variances. Second, we give new theoretical guarantees for inference after trimming. In particular, we show how to perform inference on the trimmed sub-population without requiring that our regressions converge at parametric rates. Instead, we make only fourth-root rate assumptions like those in the double machine learning literature. This result applies to conventional propensity-based trimming as well and thus may be of independent interest. Finally, we propose a bootstrap-based method for constructing simultaneously valid confidence intervals for multiple trimmed sub-populations, which are valuable for navigating the trade-off between sample size and variance reduction inherent in trimming. We validate our methods in simulation, on the 2007-2008 National Health and Nutrition Examination Survey, and on a semi-synthetic Medicare dataset and find promising results in all settings.

INFERENCE IN MILDLY EXPLOSIVE AUTOREGRESSIONS UNDER UNCONDITIONAL HETEROSKEDASTICITY

Mildly explosive autoregressions have been extensively employed in recent theoretical and applied econometric work to model the phenomenon of asset market bubbles. An important issue in this context concerns the construction of confidence intervals for the autoregressive parameter that represents the degree of explosiveness. Existing studies rely on intervals that are justified only under conditional homoskedasticity/heteroskedasticity. This paper studies the problem of constructing asymptotically valid confidence intervals in a mildly explosive autoregression where the innovations are allowed to be unconditionally heteroskedastic. The assumed variance process is general and can accommodate both deterministic and stochastic volatility specifications commonly adopted in the literature. Within this framework, we show that the standard heteroskedasticity- and autocorrelation-consistent estimate of the long-run variance converges in distribution to a nonstandard random variable that depends on nuisance parameters. Notwithstanding this result, the corresponding t-statistic is shown to still possess a standard normal limit distribution. To improve the quality of inference in small samples, we propose a dependent wild bootstrap-t procedure and establish its asymptotic validity under relatively weak conditions. Monte Carlo simulations demonstrate that our recommended approach performs favorably in finite samples relative to existing methods across a wide range of volatility specifications. Applications to international stock price indices and U.S. house prices illustrate the relevance of the advocated method in practice.

Test and Measure for Partial Mean Dependence Based on Machine Learning Methods

It is of importance to investigate the significance of a subset of covariates W for the response Y given covariates Z in regression modeling. To this end, we propose a significance test for the partial mean independence problem based on machine learning methods and data splitting. The test statistic converges to the standard Chi-squared distribution under the null hypothesis while it converges to a normal distribution under the fixed alternative hypothesis. Power enhancement and algorithm stability are also discussed. If the null hypothesis is rejected, we propose a partial Generalized Measure of Correlation (pGMC) to measure the partial mean dependence of Y given W after controlling for the nonlinear effect of Z. We present the appealing theoretical properties of the pGMC and establish the asymptotic normality of its estimator with the optimal root-N convergence rate. Furthermore, the valid confidence interval for the pGMC is also derived. As an important special case when there are no conditional covariates Z, we introduce a new test of overall significance of covariates for the response in a model-free setting. Numerical studies and real data analysis are also conducted to compare with existing approaches and to demonstrate the validity and flexibility of our proposed procedures. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Statistical Inference for Heterogeneous Treatment Effects Discovered by Generic Machine Learning in Randomized Experiments

Researchers are increasingly turning to machine learning (ML) algorithms to investigate causal heterogeneity in randomized experiments. Despite their promise, ML algorithms may fail to accurately ascertain heterogeneous treatment effects under practical settings with many covariates and small sample size. In addition, the quantification of estimation uncertainty remains a challenge. We develop a general approach to statistical inference for heterogeneous treatment effects discovered by a generic ML algorithm. We apply the Neyman’s repeated sampling framework to a common setting, in which researchers use an ML algorithm to estimate the conditional average treatment effect and then divide the sample into several groups based on the magnitude of the estimated effects. We show how to estimate the average treatment effect within each of these groups, and construct a valid confidence interval. In addition, we develop nonparametric tests of treatment effect homogeneity across groups, and rank-consistency of within-group average treatment effects. The validity of our methodology does not rely on the properties of ML algorithms because it is solely based on the randomization of treatment assignment and random sampling of units. Finally, we generalize our methodology to the cross-fitting procedure by accounting for the additional uncertainty induced by the random splitting of data.

Joint Statistical Inference for the Area under the ROC Curve and Youden Index under a Density Ratio Model

The receiver operating characteristic (ROC) curve is a valuable statistical tool in medical research. It assesses a biomarker’s ability to distinguish between diseased and healthy individuals. The area under the ROC curve (AUC) and the Youden index (J) are common summary indices used to evaluate a biomarker’s diagnostic accuracy. Simultaneously examining AUC and J offers a more comprehensive understanding of the ROC curve’s characteristics. In this paper, we utilize a semiparametric density ratio model to link the distributions of a biomarker for healthy and diseased individuals. Under this model, we establish the joint asymptotic normality of the maximum empirical likelihood estimator of (AUC,J) and construct an asymptotically valid confidence region for (AUC,J). Furthermore, we propose a new test to determine whether a biomarker simultaneously exceeds prespecified target values of AUC0 and J0 with the null hypothesis H0:AUC≤AUC0 or J≤J0 against the alternative hypothesis Ha:AUC>AUC0 and J>J0. Simulation studies and a real data example on Duchenne Muscular Dystrophy are used to demonstrate the effectiveness of our proposed method and highlight its advantages over existing methods.

Simultaneous Confidence Intervals for Partially Identified Parameters

This article extends the Imbens and Manski and Stoye confidence interval for a partially identified scalar parameter to a vector-valued parameter. The proposed method produces uniformly valid simultaneous confidence intervals for each dimension, or, equivalently, a rectangular confidence region that covers points in the identified set with a specified probability. The method applies when asymptotically normal estimates of upper and lower bounds for each dimension are available. The intervals are computationally simple and fast relative to methods based on test inversion or bootstrapped calibration, and do not suffer from the conservativity of projection-based approaches.

Double debiased transfer learning for adaptive Huber regression

AbstractThrough exploiting information from the source data to improve the fit performance on the target data, transfer learning estimations for high‐dimensional linear regression models have drawn much attention recently, but few studies focus on statistical inference and robust learning in the presence of heavy‐tailed/asymmetric errors. Using adaptive Huber regression (AHR) to achieve the bias and robustness tradeoff, in this paper we propose a robust transfer learning algorithm with high‐dimensional covariates, then construct valid confidence intervals and hypothesis tests based on the debiased lasso approach. When the transferable sources are known, a two‐step ‐penalized transfer AHR estimator is firstly proposed and the error bounds are established. To correct the biases caused by the lasso penalty, a unified debiasing framework based on the decorrelated score equations is considered to establish asymptotic normality of the debiased lasso transfer AHR estimator. Confidence intervals and hypothesis tests for each component can be constructed. When the transferable sources are unknown, a data‐driven source detection algorithm is proposed with theoretical guarantee. Numerical studies verify the performance of our proposed estimator and confidence intervals, and an application to Genotype‐Tissue Expression data is also presented.

Sufficient Conditions for Central Limit Theorems and Confidence Intervals for Randomized Quasi-Monte Carlo Methods

Randomized quasi-Monte Carlo methods have been introduced with the main purpose of yielding a computable measure of error for quasi-Monte Carlo approximations through the implicit application of a central limit theorem over independent randomizations. But to increase precision for a given computational budget, the number of independent randomizations is usually set to a small value so that a large number of points are used from each randomized low-discrepancy sequence to benefit from the fast convergence rate of quasi-Monte Carlo. While a central limit theorem has been previously established for a specific but computationally expensive type of randomization, it is also known in general that fixing the number of randomizations and increasing the length of the sequence used for quasi-Monte Carlo can lead to a non-Gaussian limiting distribution. This paper presents sufficient conditions on the relative growth rates of the number of randomizations and the quasi-Monte Carlo sequence length to ensure a central limit theorem and also an asymptotically valid confidence interval. We obtain several results based on the Lindeberg condition for triangular arrays and expressed in terms of the regularity of the integrand and the convergence speed of the quasi-Monte Carlo method. We also analyze the resulting estimator’s convergence rate.

Abstract SY02-03: Stochastic modeling of clonal evolution in carcinogenesis

Abstract In order to develop strategies that can detect cancer earlier, we need to better understand the dynamics of cancer evolution during the human lifetime. Starting from gestation, this includes measuring both the timing of early somatic mutations that lead to clonal mosaicism as well as the growth rates of mutated clones that carry higher potential for malignant transformation. Continuous observation of this evolving clonal composition across a lifetime is not feasible, therefore mathematical models are needed to integrate data from longitudinal and cross-sectional samples and infer these dynamics. Stochastic models of clonal evolution offer such a framework and provide a means for predicting individual clone trajectories into the future. Furthermore, if measurements of evolution are to be utilized clinically for early detection and patient risk stratification, faster methods to apply these models to data are required. To this end, we derived methods using coalescent theory for single-cell DNA sequencing data that enable instantaneous estimation of the growth rate of a clone along with its confidence intervals. Our tool is available in an open source R package, cloneRate, and provides the first method for constructing valid confidence intervals of growth rates analytically without having to rely on Bayesian inference techniques and complex simulations. When applying our methods to recent datasets derived from normal and neoplastic human blood cells, we quantified increased fitness effects of multi-hit driver mutations and found that higher initial clone growth rates led to decreased time to cancer diagnosis in patients. We can also use these models to estimate other important biological parameters that are difficult to measure in vivo such as bounds for total number of hematopoietic stem cells and expansion rates in early development. Ultimately, the ability to easily and quickly parameterize clonal dynamics in individual patients will benefit future early detection efforts and screening strategies for many cancer types. Citation Format: Kit Curtius. Stochastic modeling of clonal evolution in carcinogenesis [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2024; Part 2 (Late-Breaking, Clinical Trial, and Invited Abstracts); 2024 Apr 5-10; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2024;84(7_Suppl):Abstract nr SY02-03.

On Sample Size Needed for Block Bootstrap Confidence Intervals to Have Desired Coverage Rates

Block bootstrap is widely used in constructing confidence intervals for parameters estimated from stationary time series. Theoretically, the method should provide valid confidence intervals as the length of the time series goes to infinity. In practice, however, it is necessary to know how large of a finite sample is required for block bootstrap confidence intervals to work well. This study aims to answer this question in a simple simulation setting where the data are generated from a first-order autoregressive process. The empirical coverage rates of several commonly used bootstrap confidence intervals for the mean, standard deviation, and the lag-1 autocorrelation coefficient are compared. A quite large sample is found necessary for the intervals to have the right coverage rates even when estimating a simple parameter like the mean. Some block bootstrap methods could fail when estimating the lag-1 autocorrelation. It is surprising that the coverage property even deteriorates as the sample size increases with some commonly used block bootstrap confidence intervals including the percentile intervals and bias-corrected intervals. KEYWORDS: Autocorrelation; Bias-Correction; Centering; Dependent Data; Percentile; Resampling; Simulation; Time Series

Analysis of Differentially Private Synthetic Data: A Measurement Error Approach

Differentially private (DP) synthetic datasets have been receiving significant attention from academia, industry, and government. However, little is known about how to perform statistical inference using DP synthetic datasets. Naive approaches that do not take into account the induced uncertainty due to the DP mechanism will result in biased estimators and invalid inferences. In this paper, we present a class of maximum likelihood estimator (MLE)-based easy-to-implement bias-corrected DP estimators with valid asymptotic confidence intervals (CI) for parameters in regression settings, by establishing the connection between additive DP mechanisms and measurement error models. Our simulation shows that our estimator has comparable performance to the widely used sufficient statistic perturbation (SSP) algorithm in some scenarios but with the advantage of releasing a synthetic dataset and obtaining statistically valid asymptotic CIs, which can achieve better coverage when compared to the naive CIs obtained by ignoring the DP mechanism.

Resampling-based confidence intervals and bands for the average treatment effect in observational studies with competing risks

The g-formula can be used to estimate the treatment effect while accounting for confounding bias in observational studies. With regard to time-to-event endpoints, possibly subject to competing risks, the construction of valid pointwise confidence intervals and time-simultaneous confidence bands for the causal risk difference is complicated, however. A convenient solution is to approximate the asymptotic distribution of the corresponding stochastic process by means of resampling approaches. In this paper, we consider three different resampling methods, namely the classical nonparametric bootstrap, the influence function equipped with a resampling approach as well as a martingale-based bootstrap version, the so-called wild bootstrap. For the latter, three sub-versions based on differing distributions of the underlying random multipliers are examined. We set up a simulation study to compare the accuracy of the different techniques, which reveals that the wild bootstrap should in general be preferred if the sample size is moderate and sufficient data on the event of interest have been accrued. For illustration, the resampling methods are further applied to data on the long-term survival in patients with early-stage Hodgkin’s disease.

Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods

This article investigates the problem of online statistical inference of model parameters in stochastic optimization problems via the Kiefer-Wolfowitz algorithm with random search directions. We first present the asymptotic distribution for the Polyak-Ruppert-averaging type Kiefer-Wolfowitz (AKW) estimators, whose asymptotic covariance matrices depend on the distribution of search directions and the function-value query complexity. The distributional result reflects the tradeoff between statistical efficiency and function query complexity. We further analyze the choice of random search directions to minimize certain summary statistics of the asymptotic covariance matrix. Based on the asymptotic distribution, we conduct online statistical inference by providing two construction procedures of valid confidence intervals. Supplementary materials for this article are available online.

An identification and testing strategy for proxy-SVARs with weak proxies

When proxies (external instruments) used to identify target structural shocks are weak, inference in proxy-SVARs (SVAR-IVs) is nonstandard and the construction of asymptotically valid confidence sets for the impulse responses of interest requires weak-instrument robust methods. In the presence of multiple target shocks, test inversion techniques require extra restrictions on the proxy-SVAR parameters other than those implied by the proxies that may be difficult to interpret and test. We show that frequentist asymptotic inference in these situations can be conducted through Minimum Distance estimation and standard asymptotic methods if the proxy-SVAR can be identified by using ‘strong’ instruments for the non-target shocks; i.e., the shocks which are not of primary interest in the analysis. The suggested identification strategy hinges on a novel pre-test for the null of instrument relevance, based on bootstrap resampling, which is not subject to pre-testing issues. Specifically, the validity of post-test asymptotic inferences remains unaffected by the test outcomes due to an asymptotic independence result between the bootstrap and non-bootstrap statistics. The test is robust to conditionally heteroskedastic and/or zero-censored proxies, is computationally straightforward and applicable regardless of the number of shocks being instrumented. Some illustrative examples show the empirical usefulness of the suggested identification and testing strategy.

Asymptotics of K-Fold Cross Validation

This paper investigates the asymptotic distribution of the K-fold cross validation error in an i.i.d. setting. As the number of observations n goes to infinity while keeping the number of folds K fixed, the K-fold cross validation error is √ n-consistent for the expected out-of-sample error and has an asymptotically normal distribution. A consistent estimate of the asymptotic variance is derived and used to construct asymptotically valid confidence intervals for the expected out-of-sample error. A hypothesis test is developed for comparing two estimators’ expected out-of-sample errors and a subsampling procedure is used to obtain critical values. Monte Carlo simulations demonstrate the asymptotic validity of our confidence intervals for the expected out-of-sample error and investigate the size and power properties of our test. In our empirical application, we use our estimator selection test to compare the out-of-sample predictive performance of OLS, Neural Networks, and Random Forests for predicting the sale price of a domain name in a GoDaddy expiry auction.