High-dimensional Logistic Regression Research Articles

Robust adaptive LASSO in high-dimensional logistic regression

Robust adaptive LASSO in high-dimensional logistic regression

An integrated molecular risk score early in life for subsequent childhood asthma risk.

Numerous children present with early wheeze symptoms, yet solely a subgroup develops childhood asthma. Early identification of children at risk is key for clinical monitoring, timely patient-tailored treatment, and preventing chronic, severe sequelae. For early prediction of childhood asthma, we aimed to define an integrated risk score combining established risk factors with genome-wide molecular markers at birth, complemented by subsequent clinical symptoms/diagnoses (wheezing, atopic dermatitis, food allergy). Three longitudinal birth cohorts (PAULINA/PAULCHEN, n = 190 + 93 = 283, PASTURE, n = 1133) were used to predict childhood asthma (age 5-11) including epidemiological characteristics and molecular markers: genotype, DNA methylation and mRNA expression (RNASeq/NanoString). Apparent (ap) and optimism-corrected (oc) performance (AUC/R2) was assessed leveraging evidence from independent studies (Naïve-Bayes approach) combined with high-dimensional logistic regression models (LASSO). Asthma prediction with epidemiological characteristics at birth (maternal asthma, sex, farm environment) yielded an ocAUC = 0.65. Inclusion of molecular markers as predictors resulted in an improvement in apparent prediction performance, however, for optimism-corrected performance only a moderate increase was observed (upto ocAUC = 0.68). The greatest discriminate power was reached by adding the first symptoms/diagnosis (up to ocAUC = 0.76; increase of 0.08, p = .002). Longitudinal analysis of selected mRNA expression in PASTURE (cord blood, 1, 4.5, 6 years) showed that expression at age six had the strongest association with asthma and correlation of genes getting larger over time (r = .59, p < .001, 4.5-6 years). Applying epidemiological predictors alone showed moderate predictive abilities. Molecular markers from birth modestly improved prediction. Allergic symptoms/diagnoses enhanced the power of prediction, which is important for clinical practice and for the design of future studies with molecular markers.

Bayesian Inference on Hierarchical Nonlocal Priors in Generalized Linear Models

Variable selection methods with nonlocal priors have been widely studied in linear regression models, and their theoretical and empirical performances have been reported. However, the crucial model selection properties for hierarchical nonlocal priors in high-dimensional generalized linear regression have rarely been investigated. In this paper, we consider a hierarchical nonlocal prior for high-dimensional logistic regression models and investigate theoretical properties of the posterior distribution. Specifically, a product moment (pMOM) nonlocal prior is imposed over the regression coefficients with an Inverse-Gamma prior on the tuning parameter. Under standard regularity assumptions, we establish strong model selection consistency in a high-dimensional setting, where the number of covariates is allowed to increase at a sub-exponential rate with the sample size. We implement the Laplace approximation for computing the posterior probabilities, and a modified shotgun stochastic search procedure is suggested for efficiently exploring the model space. We demonstrate the validity of the proposed method through simulation studies and an RNA-sequencing dataset for stratifying disease risk.

Statistical Inference for Genetic Relatedness Based on High-Dimensional Logistic Regression

Statistical Inference for Genetic Relatedness Based on High-Dimensional Logistic Regression

A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression

A tradeoff between false discovery and true positive proportions for sparse high-dimensional logistic regression

Debiased inference for heterogeneous subpopulations in a high-dimensional logistic regression model

Due to the prevalence of complex data, data heterogeneity is often observed in contemporary scientific studies and various applications. Motivated by studies on cancer cell lines, we consider the analysis of heterogeneous subpopulations with binary responses and high-dimensional covariates. In many practical scenarios, it is common to use a single regression model for the entire data set. To do this effectively, it is critical to quantify the heterogeneity of the effect of covariates across subpopulations through appropriate statistical inference. However, the high dimensionality and discrete nature of the data can lead to challenges in inference. Therefore, we propose a novel statistical inference method for a high-dimensional logistic regression model that accounts for heterogeneous subpopulations. Our primary goal is to investigate heterogeneity across subpopulations by testing the equivalence of the effect of a covariate and the significance of the overall effects of a covariate. To achieve overall sparsity of the coefficients and their fusions across subpopulations, we employ a fused group Lasso penalization method. In addition, we develop a statistical inference method that incorporates bias correction of the proposed penalized method. To address computational issues due to the nonlinear log-likelihood and the fused Lasso penalty, we propose a computationally efficient and fast algorithm by adapting the ideas of the proximal gradient method and the alternating direction method of multipliers (ADMM) to our settings. Furthermore, we develop non-asymptotic analyses for the proposed fused group Lasso and prove that the debiased test statistics admit chi-squared approximations even in the presence of high-dimensional variables. In simulations, the proposed test outperforms existing methods. The practical effectiveness of the proposed method is demonstrated by analyzing data from the Cancer Cell Line Encyclopedia (CCLE).

On inference in high-dimensional logistic regression models with separated data

Summary Direct use of the likelihood function typically produces severely biased estimates when the dimension of the parameter vector is large relative to the effective sample size. With linearly separable data generated from a logistic regression model, the loglikelihood function asymptotes and the maximum likelihood estimator does not exist. We show that an exact analysis for each regression coefficient produces half-infinite confidence sets for some parameters when the data are separable. Such conclusions are not vacuous, but an honest portrayal of the limitations of the data. Finite confidence sets are only achievable when additional, perhaps implicit, assumptions are made. Under a notional double-asymptotic regime in which the dimension of the logistic coefficient vector increases with the sample size, the present paper considers the implications of enforcing a natural constraint on the vector of logistic transformed probabilities. We derive a relationship between the logistic coefficients and a notional parameter obtained as a probability limit of an ordinary least-squares estimator. The latter exists even when the data are separable. Consistency is ascertained under weak conditions on the design matrix.

Optimal friction matrix for underdamped Langevin sampling

We propose a procedure for optimising the friction matrix of underdamped Langevin dynamics when used for continuous time Markov Chain Monte Carlo. Starting from a central limit theorem for the ergodic average, we present a new expression of the gradient of the asymptotic variance with respect to friction matrix. In addition, we present an approximation method that uses simulations of the associated first variation/tangent process. Our algorithm is applied to a variety of numerical examples such as toy problems with tractable asymptotic variance, diffusion bridge sampling and Bayesian inference problems for high dimensional logistic regression.

FDR control and power analysis for high-dimensional logistic regression via StabKoff

FDR control and power analysis for high-dimensional logistic regression via StabKoff

Efficient Bayesian High-Dimensional Classification via Random Projection with Application to Gene Expression Data

Inspired by the impressive successes of compress sensing-based machine learning algorithms, data augmentation-based efficient Gibbs samplers for Bayesian high-dimensional classification models are developed by compressing the design matrix to a much lower dimension. Ardent care is exercised in the choice of the projection mechanism, and an adaptive voting rule is employed to reduce sensitivity to the random projection matrix. Focusing on the high-dimensional Probit regression model, we note that the naive implementation of the data augmentation-based Gibbs sampler is not robust to the presence of co-linearity in the design matrix – a setup ubiquitous in $n\lt p$ problems. We demonstrate that a simple fix based on joint updates of parameters in the latent space circumnavigates this issue. With a computationally efficient MCMC scheme in place, we introduce an ensemble classifier by creating R ($\sim 25$–50) projected copies of the design matrix, and subsequently running R classification models with the R projected design matrix in parallel. We combine the output from the R replications via an adaptive voting scheme. Our scheme is inherently parallelizable and capable of taking advantage of modern computing environments often equipped with multiple cores. The empirical success of our methodology is illustrated in elaborate simulations and gene expression data applications. We also extend our methodology to a high-dimensional logistic regression model and carry out numerical studies to showcase its efficacy.

A Conditional Bayesian Approach with Valid Inference for High Dimensional Logistic Regression

A Conditional Bayesian Approach with Valid Inference for High Dimensional Logistic Regression

Post-selection Inference of High-dimensional Logistic Regression Under Case–Control Design

Confidence sets are of key importance in high-dimensional statistical inference. Under case–control study, a popular response-selective sampling design in medical study or econometrics, we consider the confidence intervals and statistical tests for single or low-dimensional parameters in high-dimensional logistic regression model. The asymptotic properties of the resulting estimators are established under mild conditions. We also study statistical tests for testing more general and complex hypotheses of the high-dimensional parameters. The general testing procedures are proved to be asymptotically exact and have satisfactory power. Numerical studies including extensive simulations and a real data example confirm that the proposed method performs well in practical settings.

Automated NLP Extraction of Clinical Rationale for Treatment Discontinuation in Breast Cancer.

Key oncology end points are not routinely encoded into electronic medical records (EMRs). We assessed whether natural language processing (NLP) can abstract treatment discontinuation rationale from unstructured EMR notes to estimate toxicity incidence and progression-free survival (PFS). We constructed a retrospective cohort of 6,115 patients with early-stage and 701 patients with metastatic breast cancer initiating care at Memorial Sloan Kettering Cancer Center from 2008 to 2019. Each cohort was divided into training (70%), validation (15%), and test (15%) subsets. Human abstractors identified the clinical rationale associated with treatment discontinuation events. Concatenated EMR notes were used to train high-dimensional logistic regression and convolutional neural network models. Kaplan-Meier analyses were used to compare toxicity incidence and PFS estimated by our NLP models to estimates generated by manual labeling and time-to-treatment discontinuation (TTD). Our best high-dimensional logistic regression models identified toxicity events in early-stage patients with an area under the curve of the receiver-operator characteristic of 0.857 ± 0.014 (standard deviation) and progression events in metastatic patients with an area under the curve of 0.752 ± 0.027 (standard deviation). NLP-extracted toxicity incidence and PFS curves were not significantly different from manually extracted curves (P = .95 and P = .67, respectively). By contrast, TTD overestimated toxicity in early-stage patients (P < .001) and underestimated PFS in metastatic patients (P < .001). Additionally, we tested an extrapolation approach in which 20% of the metastatic cohort were labeled manually, and NLP algorithms were used to abstract the remaining 80%. This extrapolated outcomes approach resolved PFS differences between receptor subtypes (P < .001 for hormone receptor+/human epidermal growth factor receptor 2- v human epidermal growth factor receptor 2+ v triple-negative) that could not be resolved with TTD. NLP models are capable of abstracting treatment discontinuation rationale with minimal manual labeling.

Robust Variable Selection with Optimality Guarantees for High-Dimensional Logistic Regression

High-dimensional classification studies have become widespread across various domains. The large dimensionality, coupled with the possible presence of data contamination, motivates the use of robust, sparse estimation methods to improve model interpretability and ensure the majority of observations agree with the underlying parametric model. In this study, we propose a robust and sparse estimator for logistic regression models, which simultaneously tackles the presence of outliers and/or irrelevant features. Specifically, we propose the use of L0-constraints and mixed-integer conic programming techniques to solve the underlying double combinatorial problem in a framework that allows one to pursue optimality guarantees. We use our proposal to investigate the main drivers of honey bee (Apis mellifera) loss through the annual winter loss survey data collected by the Pennsylvania State Beekeepers Association. Previous studies mainly focused on predictive performance, however our approach produces a more interpretable classification model and provides evidence for several outlying observations within the survey data. We compare our proposal with existing heuristic methods and non-robust procedures, demonstrating its effectiveness. In addition to the application to honey bee loss, we present a simulation study where our proposal outperforms other methods across most performance measures and settings.

Bayesian variable selection in logistic regression with application to whole-brain functional connectivity analysis for Parkinson's disease.

Parkinson's disease is a progressive, chronic, and neurodegenerative disorder that is primarily diagnosed by clinical examinations and magnetic resonance imaging (MRI). In this paper, we propose a Bayesian model to predict Parkinson's disease employing a functional MRI (fMRI) based radiomics approach. We consider a spike and slab prior for variable selection in high-dimensional logistic regression models, and present an approximate Gibbs sampler by replacing a logistic distribution with a t-distribution. Under mild conditions, we establish model selection consistency of the induced posterior and illustrate the performance of the proposed method outperforms existing state-of-the-art methods through simulation studies. In fMRI analysis, 6216 whole-brain functional connectivity features are extracted for 50 healthy controls along with 70 Parkinson's disease patients. We apply our method to the resulting dataset and further show its benefits with a higher average prediction accuracy of 0.83 compared to other contenders based on 10 random splits. The model fitting procedure also reveals the most discriminative brain regions for Parkinson's disease. These findings demonstrate that the proposed Bayesian variable selection method has the potential to support radiological diagnosis for patients with Parkinson's disease.

Ensemble forecasting for intraday electricity prices: Simulating trajectories

Recent studies concerning the point electricity price forecasting have shown evidence that the hourly German Intraday Continuous Market is weak-form efficient. Therefore, we take a novel, advanced approach to the problem. A probabilistic forecasting of the hourly intraday electricity prices is performed by simulating trajectories in every trading window to receive a realistic ensemble to allow for more efficient intraday trading and redispatch. A generalized additive model is fitted to the price differences with the assumption that they follow a zero-inflated distribution, precisely a mixture of the Dirac and the Student’s t-distributions. Moreover, the mixing term is estimated using a high-dimensional logistic regression with lasso penalty. We model the expected value and volatility of the series using i.a. autoregressive and no-trade effects or load, wind and solar generation forecasts and accounting for the non-linearities in e.g. time to maturity. Both the in-sample characteristics and forecasting performance are analysed using a rolling window forecasting study. Multiple versions of the model are compared to several benchmark models and evaluated using probabilistic forecasting measures and significance tests. The study aims to forecast the price distribution in the German Intraday Continuous Market in the last 3 h of trading, but the approach allows for application to other continuous markets, especially in Europe. The results prove superiority of the mixture model over the benchmarks gaining the most from the modelling of the volatility. They also indicate that the introduction of XBID reduced the market volatility.

The phase transition for the existence of the maximum likelihood estimate in high-dimensional logistic regression

This paper rigorously establishes that the existence of the maximum likelihood estimate (MLE) in high-dimensional logistic regression models with Gaussian covariates undergoes a sharp “phase transition.” We introduce an explicit boundary curve $h_{\mathrm{MLE}}$, parameterized by two scalars measuring the overall magnitude of the unknown sequence of regression coefficients, with the following property: in the limit of large sample sizes $n$ and number of features $p$ proportioned in such a way that $p/n\rightarrow \kappa $, we show that if the problem is sufficiently high dimensional in the sense that $\kappa >h_{\mathrm{MLE}}$, then the MLE does not exist with probability one. Conversely, if $\kappa <h_{\mathrm{MLE}}$, the MLE asymptotically exists with probability one.

Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models

High-dimensional logistic regression is widely used in analyzing data with binary outcomes. In this article, global testing and large-scale multiple testing for the regression coefficients are considered in both single- and two-regression settings. A test statistic for testing the global null hypothesis is constructed using a generalized low-dimensional projection for bias correction and its asymptotic null distribution is derived. A lower bound for the global testing is established, which shows that the proposed test is asymptotically minimax optimal over some sparsity range. For testing the individual coefficients simultaneously, multiple testing procedures are proposed and shown to control the false discovery rate and falsely discovered variables asymptotically. Simulation studies are carried out to examine the numerical performance of the proposed tests and their superiority over existing methods. The testing procedures are also illustrated by analyzing a dataset of a metabolomics study that investigates the association between fecal metabolites and pediatric Crohn’s disease and the effects of treatment on such associations. Supplementary materials for this article are available online.

The existence of maximum likelihood estimate in high-dimensional binary response generalized linear models

Motivated by recent works on the high-dimensional logistic regression, we establish that the existence of the maximum likelihood estimate exhibits a phase transition for a wide range of generalized linear models with binary outcome and elliptical covariates. This extends a previous result of Candès and Sur who proved the phase transition for the logistic regression with Gaussian covariates. Our result reveals a rich structure in the phase transition phenomenon, which is simply overlooked by Gaussianity. The main tools for deriving the result are data separation, convex geometry and stochastic approximation. We also conduct simulation studies to corroborate our theoretical findings, and explore other features of the problem.

Structured sparse logistic regression with application to lung cancer prediction using breath volatile biomarkers.

This article is motivated by a study of lung cancer prediction using breath volatile organic compound (VOC) biomarkers, where the challenge is that the predictors include not only high-dimensional time-dependent or functional VOC features but also the time-independent clinical variables. We consider a high-dimensional logistic regression and propose two different penalties: group spline-penalty or group smooth-penalty to handle the group structures of the time-dependent variables in the model. The new methods have the advantage for the situation where the model coefficients are sparse but change smoothly within the group, compared with other existing methods such as the group lasso and the group bridge approaches. Our methods are easy to implement since they can be turned into a group minimax concave penalty problem after certain transformations. We show that our fitting algorithm possesses the descent property and leads to attractive convergence properties. The simulation studies and the lung cancer application are performed to demonstrate the accuracy and stability of the proposed approaches.