Constructing Confidence Sets Research Articles

Distributionally robust CVaR optimization for refinery integrated production–maintenance scheduling under uncertainty

In the petroleum refining industry, efficient production planning and maintenance scheduling are crucial for economic performance and operational efficiency. Moreover, the production processes face significant uncertainties stemming from market fluctuations and equipment failures. However, traditional optimization methods often treat production and maintenance independently and neglect the risk management associated with uncertainties in the production process, leading to unreliable plans and suboptimal execution. To address these issues, this paper proposes an innovative data-driven distributionally robust conditional value-at-risk (DRCVaR) method to tackle the integrated production–maintenance optimization problem under crude oil price uncertainty. By constructing confidence sets with L2 norm constraints based on historical data, our approach directly links the model’s conservatism to the amount of available data, effectively managing risk. In addition, we propose robust linear transformation to simplify the min–max nonlinear problem into a conic constraint problem, enhancing solution efficiency and ensuring better operational stability. Refinery case studies demonstrate that the proposed DRCVaR consistently achieves a practical and acceptable solution, significantly outperforming state-of-the-art approaches.

Confidence sets for intraclass correlation coefficients in test–retest curve measurements

Evaluating test–retest reliability is crucial in biomechanical research, as it validates experimental results. While methods for reliability of scalar outcome variables are well-established, methods to assess reliability of continuous curve data (such as joint angle trajectories during gait) remain less explored. This study investigates methods for constructing confidence sets for curve-level intraclass correlation coefficients (ICC), which can be expressed as either an ICC curve or an integrated ICC. Currently, no standardised guidelines exist in biomechanics for reporting curve-level ICC uncertainty. Nonparametric bootstrapping techniques are proposed for both the ICC curve’s confidence bands and the integrated ICC’s confidence intervals, and these methods are validated through Monte Carlo simulations, covering various effect sizes and curve characteristics. Additionally, these methods are applied to assess the test–retest reliability of knee kinematics in three different planes during landing of one-leg hops, where less uncertainty is observed for the ICC curve and integrated ICC in the frontal plane compared to other planes. When the entire time domain is of primary empirical interest, we recommend using a rank-based bootstrap confidence band to express ICC uncertainty, as it yields increasingly precise and valid results as the number of individuals increases, with the coverage rate approaching the correct level of 95%. When a single summary metric is of primary interest, we recommend using the integrated ICC along with a typical bootstrap confidence interval based on the normal distribution, as the coverage rate remains adequately accurate and stable at around the correct level of 95% across varying number of individuals.

Dynamic Regret of Adversarial MDPs with Unknown Transition and Linear Function Approximation

We study reinforcement learning (RL) in episodic MDPs with adversarial full-information losses and the unknown transition. Instead of the classical static regret, we adopt dynamic regret as the performance measure which benchmarks the learner's performance with changing policies, making it more suitable for non-stationary environments. The primary challenge is to handle the uncertainties of unknown transition and unknown non-stationarity of environments simultaneously. We propose a general framework to decouple the two sources of uncertainties and show the dynamic regret bound naturally decomposes into two terms, one due to constructing confidence sets to handle the unknown transition and the other due to choosing sub-optimal policies under the unknown non-stationarity. To this end, we first employ the two-layer online ensemble structure to handle the adaptation error due to the unknown non-stationarity, which is model-agnostic. Subsequently, we instantiate the framework to three fundamental MDP models, including tabular MDPs, linear MDPs and linear mixture MDPs, and present corresponding approaches to control the exploration error due to the unknown transition. We provide dynamic regret guarantees respectively and show they are optimal in terms of the number of episodes K and the non-stationarity P̄ᴋ by establishing matching lower bounds. To the best of our knowledge, this is the first work that achieves the dynamic regret exhibiting optimal dependence on K and P̄ᴋ without prior knowledge about the non-stationarity for adversarial MDPs with unknown transition.

Model Selection Path and Construction of Model Confidence Set under High-Dimensional Variables

Model selection uncertainty has drawn a lot of attention from academics recently because it significantly affects parameter estimation and prediction. Scholars are currently addressing and quantifying uncertainty in model selection by concentrating on model combining and model confidence sets. In this paper, we present a new approach for building model confidence sets, which we call AMac. We provide a theoretical lower bound on the degree of confidence in the model confidence sets that AMac has built. Furthermore, we discuss how the implementation of current model confidence set construction methods becomes difficult when dealing with high-dimensional variables. To address this problem, we suggest building model selection paths (MSP) as a solution. We develop an algorithm for building MSP and show its effectiveness by utilizing the theories of adaptive lasso and lars. We perform an extensive set of simulation experiments to compare the performances of Mac and AMac methods. According to the results, AMac is more stable when there are fluctuations in noise levels. The model confidence sets built by AMac, in particular, achieve coverage rates that are closer to the desired confidence level, especially in the presence of high noise levels. To further confirm that MSP can successfully generate model confidence sets that maintain the given confidence level as the sample size increases, we conduct extensive simulation tests with high-dimensional variables. Ultimately, we hope that the strategies and concepts discussed in this work will improve results in subsequent research on the uncertainty of model selection.

Confidence sets for a level set in linear regression.

Regression modeling is the workhorse of statistics and there is a vast literature on estimation of the regression function. It has been realized in recent years that in regression analysis the ultimate aim may be the estimation of a level set of the regression function, ie, the set of covariate values for which the regression function exceeds a predefined level, instead of the estimation of the regression function itself. The published work on estimation of the level set has thus far focused mainly on nonparametric regression, especially on point estimation. In this article, the construction of confidence sets for the level set of linear regression is considered. In particular, level upper, lower and two-sided confidence sets are constructed for the normal-error linear regression. It is shown that these confidence sets can be easily constructed from the corresponding level simultaneous confidence bands. It is also pointed out that the construction method is readily applicable to other parametric regression models where the mean response depends on a linear predictor through a monotonic link function, which include generalized linear models, linear mixed models and generalized linear mixed models. Therefore, the method proposed in this article is widely applicable. Simulation studies with both linear and generalized linear models are conducted to assess the method and real examples are used to illustrate the method.

Locally robust inference for non‐Gaussian SVAR models

All parameters in structural vector autoregressive (SVAR) models are locally identified when the structural shocks are independent and follow non‐Gaussian distributions. Unfortunately, standard inference methods that exploit such features of the data for identification fail to yield correct coverage for structural functions of the model parameters when deviations from Gaussianity are small. To this extent, we propose a locally robust semiparametric approach to conduct hypothesis tests and construct confidence sets for structural functions in SVAR models. The methodology fully exploits non‐Gaussianity when it is present, but yields correct size/coverage for local‐to‐Gaussian densities. Empirically, we revisit two macroeconomic SVAR studies where we document mixed results. For the oil price model of Kilian and Murphy (2012), we find that non‐Gaussianity can robustly identify reasonable confidence sets, whereas for the labor supply–demand model of Baumeister and Hamilton (2015) this is not the case. Moreover, these exercises highlight the importance of using weak identification robust methods to assess estimation uncertainty when using non‐Gaussianity for identification.

The HulC: confidence regions from convex hulls

Abstract We develop and analyse the HulC, an intuitive and general method for constructing confidence sets using the convex hull of estimates constructed from subsets of the data. Unlike classical methods which are based on estimating the (limiting) distribution of an estimator, the HulC is often simpler to use and effectively bypasses this step. In comparison to the bootstrap, the HulC requires fewer regularity conditions and succeeds in many examples where the bootstrap provably fails. Unlike sub-sampling, the HulC does not require knowledge of the rate of convergence of the estimators on which it is based. The validity of the HulC requires knowledge of the (asymptotic) median bias of the estimators. We further analyse a variant of our basic method, called the Adaptive HulC, which is fully data-driven and estimates the median bias using sub-sampling. We discuss these methods in the context of several challenging inferential problems which arise in parametric, semi-parametric, and non-parametric inference. Although our focus is on validity under weak regularity conditions, we also provide some general results on the width of the HulC confidence sets, showing that in many cases the HulC confidence sets have near-optimal width.

Root and community inference on the latent growth process of a network

Abstract Many statistical models for networks overlook the fact that most real-world networks are formed through a growth process. To address this, we introduce the Preferential Attachment Plus Erdős–Rényi model, where we let a random network G be the union of a preferential attachment (PA) tree T and additional Erdős–Rényi (ER) random edges. The PA tree captures the underlying growth process of a network where vertices/edges are added sequentially, while the ER component can be regarded as noise. Given only one snapshot of the final network G, we study the problem of constructing confidence sets for the root node of the unobserved growth process; the root node can be patient zero in an infection network or the source of fake news in a social network. We propose inference algorithms based on Gibbs sampling that scales to networks with millions of nodes and provide theoretical analysis showing that the size of the confidence set is small if the noise level of the ER edges is not too large. We also propose variations of the model in which multiple growth processes occur simultaneously, reflecting the growth of multiple communities; we use these models to provide a new approach to community detection.

Uniformly valid inference based on the Lasso in linear mixed models

Linear mixed models (LMMs) are suitable for clustered data and are common in biometrics, medicine, survey statistics, and many other fields. In those applications, it is essential to carry out valid inference after selecting a subset of the available variables. We construct confidence sets for the fixed effects in Gaussian LMMs that are based on Lasso-type estimators. Aside from providing confidence regions, this also allows for quantification of the joint uncertainty of both variable selection and parameter estimation in the procedure. To show that the resulting confidence sets for the fixed effects are uniformly valid over the parameter spaces of both the regression coefficients and the covariance parameters, we also prove the novel result on uniform Cramér consistency of the restricted maximum likelihood (REML) estimators of the covariance parameters. The superiority of the constructed confidence sets to naïve post-selection procedures is validated in simulations and illustrated with a study of the acid-neutralization capacity of lakes in the United States.

Archaeology of random recursive dags and Cooper-Frieze random networks

AbstractWe study the problem of finding the root vertex in large growing networks. We prove that it is possible to construct confidence sets of size independent of the number of vertices in the network that contain the root vertex with high probability in various models of random networks. The models include uniform random recursive dags and uniform Cooper-Frieze random graphs.

Finite sample inference for empirical Bayesian methods

AbstractIn recent years, empirical Bayesian (EB) inference has become an attractive approach for estimation in parametric models arising in a variety of real‐life problems, especially in complex and high‐dimensional scientific applications. However, compared to the relative abundance of available general methods for computing point estimators in the EB framework, the construction of confidence sets and hypothesis tests with good theoretical properties remains difficult and problem specific. Motivated by the Universal Inference framework, we propose a general and universal method, based on holdout likelihood ratios, and utilizing the hierarchical structure of the specified Bayesian model for constructing confidence sets and hypothesis tests that are finite sample valid. We illustrate our method through a range of numerical studies and real data applications, which demonstrate that the approach is able to generate useful and meaningful inferential statements in the relevant contexts.

Sparse confidence sets for normal mean models

Abstract In this paper, we propose a new framework to construct confidence sets for a $d$-dimensional unknown sparse parameter ${\boldsymbol \theta }$ under the normal mean model ${\boldsymbol X}\sim N({\boldsymbol \theta },\sigma ^{2}\bf{I})$. A key feature of the proposed confidence set is its capability to account for the sparsity of ${\boldsymbol \theta }$, thus named as sparse confidence set. This is in sharp contrast with the classical methods, such as the Bonferroni confidence intervals and other resampling-based procedures, where the sparsity of ${\boldsymbol \theta }$ is often ignored. Specifically, we require the desired sparse confidence set to satisfy the following two conditions: (i) uniformly over the parameter space, the coverage probability for ${\boldsymbol \theta }$ is above a pre-specified level; (ii) there exists a random subset $S$ of $\{1,...,d\}$ such that $S$ guarantees the pre-specified true negative rate for detecting non-zero $\theta _{j}$’s. To exploit the sparsity of ${\boldsymbol \theta }$, we allow the confidence interval for $\theta _{j}$ to degenerate to a single point 0 for any $j\notin S$. Under this new framework, we first consider whether there exist sparse confidence sets that satisfy the above two conditions. To address this question, we establish a non-asymptotic minimax lower bound for the non-coverage probability over a suitable class of sparse confidence sets. The lower bound deciphers the role of sparsity and minimum signal-to-noise ratio (SNR) in the construction of sparse confidence sets. Furthermore, under suitable conditions on the SNR, a two-stage procedure is proposed to construct a sparse confidence set. To evaluate the optimality, the proposed sparse confidence set is shown to attain a minimax lower bound of some properly defined risk function up to a constant factor. Finally, we develop an adaptive procedure to the unknown sparsity. Numerical studies are conducted to verify the theoretical results.

Statistical calibration for infinite many future values in linear regression: simultaneous or pointwise tolerance intervals or what else?

AbstractStatistical calibration using regression is a useful statistical tool with many applications. For confidence sets for x-values associated with infinitely many future y-values, there is a consensus in the statistical literature that the confidence sets constructed should guarantee a key property. While it is well known that the confidence sets based on the simultaneous tolerance intervals (STIs) guarantee this key property conservatively, it is desirable to construct confidence sets that satisfy this property exactly. Also, there is a misconception that the confidence sets based on the pointwise tolerance intervals (PTIs) also guarantee this property. This paper constructs the weighted simultaneous tolerance intervals (WSTIs) so that the confidence sets based on the WSTIs satisfy this property exactly if the future observations have the x-values distributed according to a known specific distribution F(⋅). Through the lens of the WSTIs, convincing counter examples are also provided to demonstrate that the confidence sets based on the PTIs do not guarantee the key property in general and so should not be used. The WSTIs have been applied to real data examples to show that the WSTIs can produce more accurate calibration intervals than STIs and PTIs.

Inference for Ranks with Applications to Mobility across Neighbourhoods and Academic Achievement across Countries

Abstract It is often desired to rank different populations according to the value of some feature of each population. For example, it may be desired to rank neighbourhoods according to some measure of intergenerational mobility or countries according to some measure of academic achievement. These rankings are invariably computed using estimates rather than the true values of these features. As a result, there may be considerable uncertainty concerning the rank of each population. In this paper, we consider the problem of accounting for such uncertainty by constructing confidence sets for the rank of each population. We consider both the problem of constructing marginal confidence sets for the rank of a particular population as well as simultaneous confidence sets for the ranks of all populations. We show how to construct such confidence sets under weak assumptions. An important feature of all of our constructions is that they remain computationally feasible even when the number of populations is very large. We apply our theoretical results to re-examine the rankings of both neighbourhoods in the U.S. in terms of intergenerational mobility and developed countries in terms of academic achievement. The conclusions about which countries do best and worst at reading, math, and science are fairly robust to accounting for uncertainty. The confidence sets for the ranking of the fifty most populous commuting zones by measures of mobility are also found to be small. These confidence sets, however, become much less informative if one includes all commuting zones, if one considers neighbourhoods at a more granular level (counties, census tracts), or if one uses movers across areas to address concerns about selection.

Dependence-Robust Confidence Intervals for Capture-Recapture Surveys.

Capture-recapture (CRC) surveys are used to estimate the size of a population whose members cannot be enumerated directly. CRC surveys have been used to estimate the number of Coronavirus Disease 2019 (COVID-19) infections, people who use drugs, sex workers, conflict casualties, and trafficking victims. When k-capture samples are obtained, counts of unit captures in subsets of samples are represented naturally by a contingency table in which one element-the number of individuals appearing in none of the samples-remains unobserved. In the absence of additional assumptions, the population size is not identifiable (i.e., point identified). Stringent assumptions about the dependence between samples are often used to achieve point identification. However, real-world CRC surveys often use convenience samples in which the assumed dependence cannot be guaranteed, and population size estimates under these assumptions may lack empirical credibility. In this work, we apply the theory of partial identification to show that weak assumptions or qualitative knowledge about the nature of dependence between samples can be used to characterize a nontrivial confidence set for the true population size. We construct confidence sets under bounds on pairwise capture probabilities using two methods: test inversion bootstrap confidence intervals and profile likelihood confidence intervals. Simulation results demonstrate well-calibrated confidence sets for each method. In an extensive real-world study, we apply the new methodology to the problem of using heterogeneous survey data to estimate the number of people who inject drugs in Brussels, Belgium.

Conformal prediction under feedback covariate shift for biomolecular design

Many applications of machine-learning methods involve an iterative protocol in which data are collected, a model is trained, and then outputs of that model are used to choose what data to consider next. For example, a data-driven approach for designing proteins is to train a regression model to predict the fitness of protein sequences and then use it to propose new sequences believed to exhibit greater fitness than observed in the training data. Since validating designed sequences in the wet laboratory is typically costly, it is important to quantify the uncertainty in the model's predictions. This is challenging because of a characteristic type of distribution shift between the training and test data that arises in the design setting-one in which the training and test data are statistically dependent, as the latter is chosen based on the former. Consequently, the model's error on the test data-that is, the designed sequences-has an unknown and possibly complex relationship with its error on the training data. We introduce a method to construct confidence sets for predictions in such settings, which account for the dependence between the training and test data. The confidence sets we construct have finite-sample guarantees that hold for any regression model, even when it is used to choose the test-time input distribution. As a motivating use case, we use real datasets to demonstrate how our method quantifies uncertainty for the predicted fitness of designed proteins and can therefore be used to select design algorithms that achieve acceptable tradeoffs between high predicted fitness and low predictive uncertainty.

Joint non-parametric estimation of mean and auto-covariances for Gaussian processes

Gaussian processes that can be decomposed into a smooth mean function and a stationary autocorrelated noise process are considered and a fully automatic nonparametric method to simultaneous estimation of mean and auto-covariance functions of such processes is developed. The proposed empirical Bayes approach is data-driven, numerically efficient, and allows for the construction of confidence sets for the mean function. Performance is demonstrated in simulations and real data analysis. The method is implemented in the R package eBsc.1

Statistical Uncertainty in the Ranking of Journals and Universities

Economists are obsessed with rankings of institutions, journals, or scholars according to the value of some feature of interest. These rankings are invariably computed using estimates rather than the true values of such features. As a result, there may be considerable uncertainty concerning the ranks. In this paper, we consider the problem of accounting for such uncertainty by constructing confidence sets for the ranks. We consider the problem of constructing marginal confidence sets for the rank of, say, a particular journal as well as simultaneous confidence sets for the ranks of all journals.

Estimation and inference on treatment effects under treatment-based sampling designs

Summary Causal inference in a programme evaluation setting faces the problem of external validity when the treatment effect in the target population is different from the treatment effect identified from the population of which the sample is representative. This paper focuses on a situation where such discrepancy arises by a stratified sampling design based on the individual treatment status and other characteristics. In such settings, the design probability is known from the sampling design but the target population depends on the underlying population share vector, which is often unknown, and, except for special cases, the treatment effect parameters are not identified. In this paper we propose a method of constructing confidence sets that are valid for a given range of population shares. When a benchmark population share vector and a corresponding estimator of a treatment effect parameter are given, we develop a method to discover the scope of external validity with familywise error rate control. Finally, we derive an optimal sampling design that minimizes the semiparametric efficiency bound given a population share associated with a target population. We provide Monte Carlo simulation results and an empirical application to demonstrate the usefulness of our proposals.

Simple Adaptive Size-Exact Testing for Full-Vector and Subvector Inference in Moment Inequality Models

Abstract We propose a simple test for moment inequalities that has exact size in normal models with known variance and has uniformly asymptotically exact size under asymptotic normality. The test compares the quasi-likelihood ratio statistic to a chi-squared critical value, where the degree of freedom is the rank of the inequalities that are active in finite samples. The test requires no simulation and thus is computationally fast and especially suitable for constructing confidence sets for parameters by test inversion. It uses no tuning parameter for moment selection and yet still adapts to the slackness of the moment inequalities. Furthermore, we show how the test can be easily adapted to inference on subvectors in the common empirical setting of conditional moment inequalities with nuisance parameters entering linearly. User-friendly Matlab code to implement the test is provided.