Post-selection Inference Research Articles

Post-selection Inference in Multiverse Analysis (PIMA): An Inferential Framework Based on the Sign Flipping Score Test.

When analyzing data, researchers make some choices that are either arbitrary, based on subjective beliefs about the data-generating process, or for which equally justifiable alternative choices could have been made. This wide range of data-analytic choices can be abused and has been one of the underlying causes of the replication crisis in several fields. Recently, the introduction of multiverse analysis provides researchers with a method to evaluate the stability of the results across reasonable choices that could be made when analyzing data. Multiverse analysis is confined to a descriptive role, lacking a proper and comprehensive inferential procedure. Recently, specification curve analysis adds an inferential procedure to multiverse analysis, but this approach is limited to simple cases related to the linear model, and only allows researchers to infer whether at least one specification rejects the null hypothesis, but not which specifications should be selected. In this paper, we present a Post-selection Inference approach to Multiverse Analysis (PIMA) which is a flexible and general inferential approach that considers for all possible models, i.e., the multiverse of reasonable analyses. The approach allows for a wide range of data specifications (i.e., preprocessing) and any generalized linear model; it allows testing the null hypothesis that a given predictor is not associated with the outcome, by combining information from all reasonable models of multiverse analysis, and provides strong control of the family-wise error rate allowing researchers to claim that the null hypothesis can be rejected for any specification that shows a significant effect. The inferential proposal is based on a conditional resampling procedure. We formally prove that the Type I error rate is controlled, and compute the statistical power of the test through a simulation study. Finally, we apply the PIMA procedure to the analysis of a real dataset on the self-reported hesitancy for the COronaVIrus Disease 2019 (COVID-19) vaccine before and after the 2020 lockdown in Italy. We conclude with practical recommendations to be considered when implementing the proposed procedure.

Exact post-selection inference for adjusted R squared selection

Post-selection inference is developed for regression coefficients selected by the widely used model selection technique of adjusted R2. Selective inference deals with the selection aspect by conditioning inference on the model selection event. In linear models, we obtain exact post-selection inference in finite samples. Extensions to logistic regression models are discussed. A simulation study illustrates the exact type I error control, unlike classical inference. Our tests provide higher power than data splitting approaches.

Parametric programming-based approximate selective inference for adaptive lasso, adaptive elastic net and group lasso

ABSTRACT Conducting model selection on data gives rise to selection uncertainty which, when ignored, invalidates subsequent classical inference which assumes that the model is given before the analysis and is in all its aspects correctly specified. In selective inference, the randomness induced by selection is dealt with by conditioning confidence intervals and p-values on the subspace of the data which leads to the same model selection as the observed data. The main challenge is the characterization of this selection event. We develop an algorithm for conducting approximate post-selection inference for parameters after model selection events which may not be characterizable as polyhedrons. We apply this on the adaptive lasso, the adaptive elastic net and the group lasso. We conduct experiments on simulated and real data, illustrating that the algorithm can both successfully control the false-positive rate and is computationally efficient.

On selection and conditioning in multiple testing and selective inference

Summary We investigate a class of methods for selective inference that condition on a selection event. Such methods follow a two-stage process. First, a data-driven collection of hypotheses is chosen from some large universe of hypotheses. Subsequently, inference takes place within this data-driven collection, conditioned on the information that was used for the selection. Examples of such methods include basic data splitting as well as modern data-carving methods and post-selection inference methods for lasso coefficients based on the polyhedral lemma. In this article, we take a holistic view of such methods, considering the selection, conditioning and final error control steps together as a single method. From this perspective, we demonstrate that multiple testing methods defined directly on the full universe of hypotheses are always at least as powerful as selective inference methods based on selection and conditioning. This result holds true even when the universe is potentially infinite and only implicitly defined, such as in the case of data splitting. We provide general theory and intuition before investigating in detail several case studies where a shift to a nonselective or unconditional perspective can yield a power gain.

Post-selection inference via algorithmic stability

Post-selection inference via algorithmic stability

Narrowest Significance Pursuit: Inference for Multiple Change-Points in Linear Models

We propose Narrowest Significance Pursuit (NSP), a general and flexible methodology for automatically detecting localized regions in data sequences which each must contain a change-point (understood as an abrupt change in the parameters of an underlying linear model), at a prescribed global significance level. NSP works with a wide range of distributional assumptions on the errors, and guarantees important stochastic bounds which directly yield exact desired coverage probabilities, regardless of the form or number of the regressors. In contrast to the widely studied “post-selection inference” approach, NSP paves the way for the concept of “post-inference selection.” An implementation is available in the R package nsp. Supplementary materials for this article are available online.

On the Use of Minimum Penalties in Statistical Learning

Modern multivariate machine learning and statistical methodologies estimate parameters of interest while leveraging prior knowledge of the association between outcome variables. The methods that do allow for estimation of relationships do so typically through an error covariance matrix in multivariate regression which does not generalize to other types of models. In this article we proposed the MinPen framework to simultaneously estimate regression coefficients associated with the multivariate regression model and the relationships between outcome variables using common assumptions. The MinPen framework uses a novel penalty based on the minimum function to simultaneously detect and exploit relationships between responses. An iterative algorithm is proposed as a solution to the nonconvex optimization. Theoretical results such as high-dimensional convergence rates, model selection consistency, and a framework for post selection inference are provided. We extend the proposed MinPen framework to other exponential family loss functions, with a specific focus on multiple binomial responses. Tuning parameter selection is also addressed. Finally, simulations and two data examples are presented to show the finite sample properties of this framework. Supplemental material providing proofs, additional simulations, code, and datasets are available online.

Density forecasting of conjunctivitis burden using high-dimensional environmental time series data

As one of the most common eye conditions being presented at clinics, acute conjunctivitis puts substantial strain on primary health resources. To reduce this public health burden, it is important to forecast and provide forward guidance to policymakers by estimating conjunctivitis trends, taking into account factors which influence transmission. Using a high-dimensional set of ambient air pollution and meteorological data, this study describes new approaches to point and probabilistic forecasting of conjunctivitis burden which can be readily translated to other infectious diseases. Over the period of 2012 – 2022, we show that simple models without environmental data provided better point forecasts but the more complex models which optimized predictive accuracy and combined multiple predictors demonstrated superior density forecast performance. These results were shown to be consistent over periods with and without structural breaks in transmission. Furthermore, ecological analysis using post-selection inference showed that increases in SO2, O3 surface concentration and total precipitation were associated to increased conjunctivitis attendance. The methods proposed can provide rich and informative forward guidance for outbreak preparedness and help guide healthcare resource planning in both stable periods of transmission and periods where structural breaks in data occur.

Rotation to Sparse Loadings Using L^p Losses and Related Inference Problems

Researchers have widely used exploratory factor analysis (EFA) to learn the latent structure underlying multivariate data. Rotation and regularised estimation are two classes of methods in EFA that they often use to find interpretable loading matrices. In this paper, we propose a new family of oblique rotations based on component-wise L^p loss functions (0 < ple 1) that is closely related to an L^p regularised estimator. We develop model selection and post-selection inference procedures based on the proposed rotation method. When the true loading matrix is sparse, the proposed method tends to outperform traditional rotation and regularised estimation methods in terms of statistical accuracy and computational cost. Since the proposed loss functions are nonsmooth, we develop an iteratively reweighted gradient projection algorithm for solving the optimisation problem. We also develop theoretical results that establish the statistical consistency of the estimation, model selection, and post-selection inference. We evaluate the proposed method and compare it with regularised estimation and traditional rotation methods via simulation studies. We further illustrate it using an application to the Big Five personality assessment.

Hybrid confidence intervals for informative uniform asymptotic inference after model selection

Abstract I propose a new type of confidence interval for correct asymptotic inference after using data to select a model of interest without assuming any model is correctly specified. This hybrid confidence interval is constructed by combining techniques from the selective inference and post-selection inference literatures to yield a short confidence interval across a wide range of data realizations. I show that hybrid confidence intervals have correct asymptotic coverage, uniformly over a large class of probability distributions that do not bound scaled model parameters. I illustrate the use of these confidence intervals in the problem of inference after using the lasso objective function to select a regression model of interest and provide evidence of their desirable length and coverage properties in small samples via a set of Monte Carlo experiments that entail a variety of different data distributions as well as an empirical application to the predictors of diabetes disease progression.

High-Dimensional Data Bootstrap

This article reviews recent progress in high-dimensional bootstrap. We first review high-dimensional central limit theorems for distributions of sample mean vectors over the rectangles, bootstrap consistency results in high dimensions, and key techniques used to establish those results. We then review selected applications of high-dimensional bootstrap: construction of simultaneous confidence sets for high-dimensional vector parameters, multiple hypothesis testing via step-down, postselection inference, intersection bounds for partially identified parameters, and inference on best policies in policy evaluation. Finally, we also comment on a couple of future research directions.

Forecasting upper respiratory tract infection burden using high-dimensional time series data and forecast combinations.

Upper respiratory tract infections (URTIs) represent a large strain on primary health resources. To mitigate URTI transmission and public health burdens, it is important to pre-empt and provide forward guidance on URTI burden, while taking into account various facets which influence URTI transmission. This is so that appropriate public health measures can be taken to mitigate strain on primary care resources. This study describes a new approach to forecasting URTIs which can be used for national public health resource planning. Specifically, using environmental and disease data comprising more than 1000 dimensions, we developed sub-models which optimizes model explainability, in-sample model fit, predictive accuracy and combines many weaker predictors over a 2-month time horizon to generate direct, point forecasts over a 1-8 week ahead forecast horizon. Predictive performance was evaluated using rolling out-of-sample forecast assessment within both periods with/without structural breaks in transmission over the period of 2012-2022. We showed that forecast combinations of 5 other forecasting models had better and more consistent predictive performance than other modelling approaches, over periods with and without structural breaks in transmission dynamics. Furthermore, epidemiological analysis on high dimensional data was enabled using post-selection inference, to show the dynamic association between lower temperature, increases in past relative humidity and absolute humidity and increased URTIs attendance. The methods proposed can be used for outbreak preparedness and guide healthcare resource planning, in both stable periods of transmission and periods where structural breaks in data occur.

Machine learning for US cross-industry return predictability under information uncertainty

This paper investigates the association between industry information uncertainty and cross-industry return predictability using machine learning in a general predictive regression framework. We show that controlling for post-selection inference and performing multiple tests improves the in-sample predictive performance of cross-industry return predictability in industries characterized by high uncertainty. Ordinary least squares post-least absolute shrinkage and selection operator models incorporating lagged industry information uncertainty for the financial and commodity industries are critical to improving prediction performance. Furthermore, in-sample industry return forecasts establish heterogeneous predictability over US industries, in which excess returns are more predictable in sectors with medium or low uncertainty.

Splitting strategies for post-selection inference

Summary We consider the problem of providing valid inference for a selected parameter in a sparse regression setting. It is well known that classical regression tools can be unreliable in this context because of the bias generated in the selection step. Many approaches have been proposed in recent years to ensure inferential validity. In this article we consider a simple alternative to data splitting based on randomizing the response vector, which allows for higher selection and inferential power than the former, and is applicable with an arbitrary selection rule. We perform a theoretical and empirical comparison of the two methods and derive a central limit theorem for the randomization approach. Our investigations show that the gain in power can be substantial.

Dissecting the impact of molecular T-cell HLA mismatches in kidney transplant failure: A retrospective cohort study.

Kidney transplantation is the optimal treatment in end-stage kidney disease, but de-novo donor specific antibody development continues to negatively impact patients undergoing kidney transplantation. One of the recent advances in solid organ transplantation has been the definition of molecular mismatching between donors and recipients' Human Leukocyte Antigens (HLA). While not fully integrated in standard clinical care, cumulative molecular mismatch at the level of eplets (EMM) as well as the PIRCHE-II score have shown promise in predicting transplant outcomes. In this manuscript, we sought to study whether certain T-cell molecular mismatches (TcEMM) were highly predictive of death-censored graft failure (DCGF). We studied a retrospective cohort of kidney donor:recipient pairs from the Scientific Registry of Transplant Recipients (2000-2015). Allele level HLA-A, B, C, DRB1 and DQB1 types were imputed from serologic types using the NMDP algorithm. TcEMMs were then estimated using the PIRCHE-II algorithm. Multivariable Accelerated Failure Time (AFT) models assessed the association between each TcEMM and DCGF. To discriminate between TcEMMs most predictive of DCGF, we fit multivariable Lasso penalized regression models. We identified co-expressed TcEMMs using weighted correlation network analysis (WGCNA). Finally, we conducted sensitivity analyses to address PIRCHE and IMGT/HLA version updates. A total of 118,309 donor:recipient pairs meeting the eligibility criteria were studied. When applying the PIRCHE-II algorithm, we identified 1,935 distinct TcEMMs at the population level. A total of 218 of the observed TcEMM were independently associated with DCGF by AFT models. The Lasso penalized regression model with post selection inference identified a smaller subset of 86 TcEMMs (56 and 30 TcEMM derived from HLA Class I and II, respectively) to be highly predictive of DCGF. Of the observed TcEMM, 38.14% appeared as profiles of highly co-expressed TcEMMs. In addition, sensitivity analyses identified that the selected TcEMM were congruent across IMGT/HLA versions. In this study, we identified subsets of TcEMMs highly predictive of DCGF and profiles of co-expressed mismatches. Experimental verification of these TcEMMs determining immune responses and how they may interact with EMM as predictors of transplant outcomes would justify their consideration in organ allocation schemes and for modifying immunosuppression regimens.

Evaluating methods for Lasso selective inference in biomedical research: a comparative simulation study

BackgroundVariable selection for regression models plays a key role in the analysis of biomedical data. However, inference after selection is not covered by classical statistical frequentist theory, which assumes a fixed set of covariates in the model. This leads to over-optimistic selection and replicability issues.MethodsWe compared proposals for selective inference targeting the submodel parameters of the Lasso and its extension, the adaptive Lasso: sample splitting, selective inference conditional on the Lasso selection (SI), and universally valid post-selection inference (PoSI). We studied the properties of the proposed selective confidence intervals available via R software packages using a neutral simulation study inspired by real data commonly seen in biomedical studies. Furthermore, we present an exemplary application of these methods to a publicly available dataset to discuss their practical usability.ResultsFrequentist properties of selective confidence intervals by the SI method were generally acceptable, but the claimed selective coverage levels were not attained in all scenarios, in particular with the adaptive Lasso. The actual coverage of the extremely conservative PoSI method exceeded the nominal levels, and this method also required the greatest computational effort. Sample splitting achieved acceptable actual selective coverage levels, but the method is inefficient and leads to less accurate point estimates.The choice of inference method had a large impact on the resulting interval estimates, thereby necessitating that the user is acutely aware of the goal of inference in order to interpret and communicate the results.ConclusionsDespite violating nominal coverage levels in some scenarios, selective inference conditional on the Lasso selection is our recommended approach for most cases. If simplicity is strongly favoured over efficiency, then sample splitting is an alternative. If only few predictors undergo variable selection (i.e. up to 5) or the avoidance of false positive claims of significance is a concern, then the conservative approach of PoSI may be useful. For the adaptive Lasso, SI should be avoided and only PoSI and sample splitting are recommended. In summary, we find selective inference useful to assess the uncertainties in the importance of individual selected predictors for future applications.

Weighted LASSO variable selection for the analysis of FTIR spectra applied to the prediction of engine oil degradation

The aim of this work is to quantify the relationship between different methods of artificial oil alteration as well as engine oils collected from a passenger car using FTIR (Fourier-transform infrared) spectroscopic data and chemometric methods. We propose a comprehensive procedure for the analysis of FTIR spectra: First, a reconstruction error based pre-processing to filter non-informative variables is introduced, then simultaneous variable selection and parameter estimation using the (weighted) LASSO is performed. Eventually, post-selection inference is applied to derive confidence intervals for the selected model coefficients. The proposed pre-processing methods do not rely on manual selection of FTIR absorption bands suitable for analysis but perform filtering of non-informative variables objectively. With weighted LASSO, experts’ knowledge can be integrated with the model. This pipeline for the analysis of FTIR spectroscopic data is demonstrated and validated on a real-world dataset including series of FTIR spectra of used and artificially altered engine oils.

Mixed-effect models with trees

Tree-based regression models are a class of statistical models for predicting continuous response variables when the shape of the regression function is unknown. They naturally take into account both non-linearities and interactions. However, they struggle with linear and quasi-linear effects and assume iid data. This article proposes two new algorithms for jointly estimating an interpretable predictive mixed-effect model with two components: a linear part, capturing the main effects, and a non-parametric component consisting of three trees for capturing non-linearities and interactions among individual-level predictors, among cluster-level predictors or cross-level. The first proposed algorithm focuses on prediction. The second one is an extension which implements a post-selection inference strategy to provide valid inference. The performance of the two algorithms is validated via Monte Carlo studies. An application on INVALSI data illustrates the potentiality of the proposed approach.

Moving beyond sub-Gaussianity in high-dimensional statistics: applications in covariance estimation and linear regression

Abstract Concentration inequalities form an essential toolkit in the study of high-dimensional statistical methods. Most of the relevant statistics literature in this regard is, however, based on the assumptions of sub-Gaussian or sub-exponential random variables/vectors. In this paper, we first bring together, through a unified exposition, various probabilistic inequalities for sums of independent random variables under much more general exponential type (namely sub-Weibull) tail assumptions. These results extract a part sub-Gaussian tail behavior of the sum in finite samples, matching the asymptotics governed by the central limit theorem, and are compactly represented in terms of a new Orlicz quasi-norm—the Generalized Bernstein–Orlicz norm—that typifies such kind of tail behaviors. We illustrate the usefulness of these inequalities through the analysis of four fundamental problems in high-dimensional statistics. In the first two problems, we study the rate of convergence of the sample covariance matrix in terms of the maximum elementwise norm and the maximum $k$-sub-matrix operator norm that are key quantities of interest in bootstrap procedures and high-dimensional structured covariance matrix estimation, as well as in high-dimensional and post-selection inference. The third example concerns the restricted eigenvalue condition, required in high-dimensional linear regression, which we verify for all sub-Weibull random vectors through a unified analysis, and also prove a more general result related to restricted strong convexity in the process. In the final example, we consider the Lasso estimator for linear regression and establish its rate of convergence to be generally $\sqrt{k\log p/n}$, for $k$-sparse signals, under much weaker than usual tail assumptions (on the errors as well as the covariates), while also allowing for misspecified models and both fixed and random design. To our knowledge, these are the first such results for Lasso obtained in this generality. The common feature in all our results over all the examples is that the convergence rates under most exponential tails match the usual (optimal) ones obtained under sub-Gaussian assumptions. Finally, we also establish some complementary results on analogous tail bounds for the suprema of empirical processes indexed by sub-Weibull variables. All our results are finite samples.

Optimal finite sample post-selection confidence distributions in generalized linear models

Uniformly most powerful confidence distributions are obtained for parameters in selected models of the exponential family. A conditioning on the selection event as well as on the sufficient statistics of nuisance parameters guarantees valid post-selection inference. Optimal confidence intervals are obtained directly from the confidence distribution without requiring an inversion of pivotal quantities. Simulations showcase that the method works also when all models are misspecified.