Irrelevant Predictors Research Articles

Efficient estimation and selection for regularized dynamic logistic regression

In various data science applications, the relationship between predictor variables and the response is dynamic in the sense that the corresponding model parameters are varying coefficients. Estimation and variable selection for such dynamic models are challenging with a large number of parameters and complex optimization. In this work, we propose a regularized dynamic logistic regression for efficient variable selection and model estimation. The proposed method considers a combination of fused and group regularization to estimate varying effects of important predictors on responses in the presence of irrelevant predictors. Specifically, we select the important variable with dynamic impact on responses through the selection of the entire group of piecewise constant functions for parameters, which can characterize dynamic impacts of predictor variables. Moreover, we develop an efficient algorithm based on the alternating direction method of multipliers for parameter estimation. The performance of the proposed method is evaluated by both simulations and real case studies.

P806 Development of a clinical model to predict secondary non-response of anti-TNFα treatment on Crohn's disease

Abstract Background Secondary non-response to anti-tumor necrosis factor-α (including Infliximab and Adalimumab) in patients with Crohn’s disease (CD) is common. Prediction of secondary non-response is essential for treatment decision for individual patients. Methods The clinical model was constructed in a training cohort. Clinical features, serological parameters, genetic biomarkers etc. were obtained as predictors after anti- TNFα therapy. Univariate analysis was performed to eliminate irrelevant predictors with secondary non-response. A logistic regression model was developed using genetic biomarkers and predictors with P < 0.1 in the univariate analysis. Finally, the model was validated with reference to its calibration and clinical decision curve. Results In a univariate analysis, infliximab antibodies,white blood cell(WBC), albumin(Alb), previous use of immunosuppressant, switching to other drugs, immunological preparation were found associated with secondary non-response to IFX. Logistic regression analysis with stepwise backward selection P < 0.01 and a clinical model was developed based on the above predictors. Conclusion The study presents a model to predict secondary non-response to anti-TNFα in CD. This model can be used to help clinicians adjust the therapeutic strategy for CD patients.

From statistical inference to machine learning: A paradigm shift in contemporary cardiovascular pharmacotherapy.

Heart failure with reduced ejection fraction (HFrEF) poses significant challenges for clinicians and researchers, owing to its multifaceted aetiology and complex treatment regimens. In light of this, artificial intelligence methods offer an innovative approach to identifying relationships within complex clinical datasets. Our study aims to explore the potential for machine learning algorithms to provide deeper insights into datasets of HFrEF patients. To this end, we analysed a cohort of 386 HFrEF patients who had been initiated on sodium-glucose co-transporter-2 inhibitor treatment and had completed a minimum of a 6-month follow-up. In traditional frequentist statistical analyses, patients receiving the highest doses of beta-blockers (BBs) (chi-square test, P = .036) and those newly initiated on sacubitril-valsartan (chi-square test, P = .023) showed better outcomes. However, none of these pharmacological features stood out as independent predictors of improved outcomes in the Cox proportional hazards model. In contrast, when employing eXtreme Gradient Boosting (XGBoost) algorithms in conjunction with the data using Shapley additive explanations (SHAP), we identified several models with significant predictive power. The XGBoost algorithm inherently accommodates non-linear distribution, multicollinearity and confounding. Within this framework, pharmacological categories like 'newly initiated treatment with sacubitril/valsartan' and 'BB dose escalation' emerged as strong predictors of long-term outcomes. In this manuscript, we not only emphasize the strengths of this machine learning approach but also discuss its potential limitations and the risk of identifying statistically significant yet clinically irrelevant predictors.

Kernel Cox partially linear regression: Building predictive models for cancer patients' survival.

Wide heterogeneity exists in cancer patients' survival, ranging from a few months to several decades. To accurately predict clinical outcomes, it is vital to build an accurate predictive model that relates the patients' molecular profiles with the patients' survival. With complex relationships between survival and high-dimensional molecular predictors, it is challenging to conduct nonparametric modeling and irrelevant predictors removing simultaneously. In this article, we build a kernel Cox proportional hazards semi-parametric model and propose a novel regularized garrotized kernel machine (RegGKM) method to fit the model. We use the kernel machine method to describe the complex relationship between survival and predictors, while automatically removing irrelevant parametric and nonparametric predictors through a LASSO penalty. An efficient high-dimensional algorithm is developed for the proposed method. Comparison with other competing methods in simulation shows that the proposed method always has better predictive accuracy. We apply this method to analyze a multiple myeloma dataset and predict the patients' death burden based on their gene expressions. Our results can help classify patients into groups with different death risks, facilitating treatment for better clinical outcomes.

Penetrating sporadic return predictability

Return predictability has been one of the central research questions in finance for many decades. This paper proposes a predictive regression with multiple structural changes to capture the sporadic predictive ability of potential predictors for the return series. An adaptive group Lasso procedure, augmented with a forward regression for break screening, is adopted to efficiently and consistently identify the structural breaks in the predictive regression, with predictors exhibiting low signal strength and heterogeneous degrees of persistence. To enhance the prediction accuracy, adaptive Lasso is further used to eliminate the irrelevant predictors and is shown to achieve the oracle property. Simulation studies demonstrate the effectiveness of the proposed methods in break detection and predictor selection, and further show that ignoring structural breaks could abate predictability. The application to predicting U.S. equity premium illustrates the practical merits of our methodology in revealing return predictability that changes over time.

What result in older persons’ acceptance of electric bikes in developing countries? An empirical evidence in Hanoi, Vietnam

The elderly are a potential segment of electric bicycles (EBs); however, so far the understanding of the factors contributing to the choice of this mode for older persons in developing countries has been rather limited – particularly compared to that in developed countries. The current study aims at exploring people’s intention to accept EB in later life in Hanoi, Vietnam using the data from 360 older persons (at least 55 years old) to empirically analyse a conceptual framework formulated based on the Model of Goal-Directed Behaviour (MGDB) and the Theory of Planned Behaviour (TPB). Desire is found to be the major contributor to intention, which is found to be significantly higher for younger and employed respondents. Desire is facilitated by subjective norms and attitude but is deterred by perceived crash risk. Perceived behavioural control and descriptive norms are irrelevant predictors. This study is one of the first research on the elderly’s intention to ride an EB in low- and middle-income countries.

Feature screening for ultrahigh-dimensional binary classification via linear projection

<abstract><p>Linear discriminant analysis (LDA) is one of the most widely used methods in discriminant classification and pattern recognition. However, with the rapid development of information science and technology, the dimensionality of collected data is high or ultrahigh, which causes the failure of LDA. To address this issue, a feature screening procedure based on the Fisher's linear projection and the marginal score test is proposed to deal with the ultrahigh-dimensional binary classification problem. The sure screening property is established to ensure that the important features could be retained and the irrelevant predictors could be eliminated. The finite sample properties of the proposed procedure are assessed by Monte Carlo simulation studies and a real-life data example.</p></abstract>

Bayesian Survival Tree Ensembles with Submodel Shrinkage

We consider Bayesian nonparametric estimation of a survival time subject to right-censoring in the presence of potentially high-dimensional predictors. We argue that several approaches, such as random survival forests and existing Bayesian nonparametric approaches, possess several drawbacks, including: computational difficulties; lack of known theoretical properties; and ineffectiveness at filtering out irrelevant predictors. We propose two models based on the Bayesian additive regression trees (BART) framework. The first, Modulated BART (MBART), is fully-nonparametric and models the failure time as the first occurrence of a non-homogeneous Poisson process. The second, CoxBART, uses a Bayesian implementation of Cox’s partial likelihood. These models are adapted to high-dimensional predictors, have default prior specifications, and require simple modifications of existing BART methods to implement. We show the effectiveness of these methods on simulated and benchmark datasets. We also establish that, for a simplified variant of MBART, the posterior distribution contracts at a near-minimax optimal rate in a high-dimensional sparse asymptotic regime.

A Scalable Hierarchical Lasso for Gene–Environment Interactions

We describe a regularized regression model for the selection of gene–environment (G × E) interactions. The model focuses on a single environmental exposure and induces a main-effect-before-interaction hierarchical structure. We propose an efficient fitting algorithm and screening rules that can discard large numbers of irrelevant predictors with high accuracy. We present simulation results showing that the model outperforms existing joint selection methods for (G × E) interactions in terms of selection performance, scalability and speed, and provide a real data application. Our implementation is available in the gesso R package. Supplementary materials for this article are available online.

Bayesian two-part modeling of phytoplankton biomass and occurrence

Phytoplankton biomass data often involve zero outcomes preventing a description by continuous distributions with positive support such as the lognormal distribution commonly used to describe ecological data. Two usual solutions: ignoring the zeroes and adding a small positive number to all outcomes, induce bias and reduce predictive power. To address these shortcomings, we design a Bayesian two-part model with a binary component for presence or absence and a continuous component involving a lognormal model for non-zero biomass. We specify two equations relating species-specific occurrence probabilities and expected log-biomasses when present to potential covariates, with spike-and-slab priors imposed on linear effects to selectively discard the irrelevant predictors. We analyze the biomass data of 74 phytoplankton (57 diatoms and 17 dinoflagellates) recorded weekly at Station L4 (Western English Channel, UK) between April 2003 and December 2009, along with measurements of abiotic covariates. Our results disclose different combinations of environmental predictors for the occurrence and the biomass of individual species. Overall, the occurrence of dinoflagellates is associated with higher temperature and irradiance levels compared to diatoms, with virtually no dependence on nutrient concentrations. Irradiance emerges as the key predictor of biomass when species are present. Optimum temperatures for biomass accumulation and temperature sensitivities vary widely among and within functional types. Compared to one-stage models based on usual zero-handling approaches, our two-part model stands out with higher prediction accuracy. The two-part modeling approach provides a valuable framework for decoupling the predictors of species occurrence and abundance from observational data.

Evaluating the Performance of Feature Selection Methods Using Huge Big Data: A Monte Carlo Simulation Approach

In this article, we compare autometrics and machine learning techniques including Minimax Concave Penalty (MCP), Elastic Smoothly Clipped Absolute Deviation (E-SCAD), and Adaptive Elastic Net (AEnet). For simulation experiments, three kinds of scenarios are considered by allowing the multicollinearity, heteroscedasticity, and autocorrelation conditions with varying sample sizes and the varied number of covariates. We found that all methods show improved their performance for a large sample size. In the presence of low and moderate multicollinearity and low and moderate autocorrelation, the considered methods retain all relevant variables. However, for low and moderate multicollinearity, excluding AEnet, all methods keep many irrelevant predictors as well. In contrast, under low and moderate autocorrelation, along with AEnet, the Autometrics retain less irrelevant predictors. Considering the case of extreme multicollinearity, AEnet retains more than 93 percent correct variables with an outstanding gauge (zero percent). However, the potency of remaining techniques, specifically MCP and E-SCAD, tends towards unity with augmenting sample size but capturing massive irrelevant predictors. Similarly, in case of high autocorrelation, E-SCAD has shown good performance in the selection of relevant variables for a small sample, while in gauge, Autometrics and AEnet are performed better and often retained less than 5 percent irrelevant variables. In the presence of heteroscedasticity, all techniques often hold all relevant variables but also suffer from overspecification problems except AEnet and Autometrics which circumvent the irrelevant predictors and establish the true model precisely. For an empirical application, we take into account the workers’ remittance data for Pakistan along its twenty-seven determinants spanning from 1972 to 2020 for Pakistan. The AEnet selected thirteen relevant covariates of workers’ remittance while E-SCAD and MCP suffered from an overspecification problem. Hence, the policymakers and practitioners should focus on the relevant variables selected by AEnet to improve workers' remittance in the case of Pakistan. In this regard, the Pakistan government has devised policies that make it easy to transfer remittances legally and mitigate the cost of transferring remittances from abroad. The AEnet approach can help policymakers arrive at relevant variables in the presence of a huge set of covariates, which in turn produce accurate predictions.

High leverage points and vertical outliers resistant model selection in regression

It is necessary to consider only relevant predictor variables for prediction purpose because irrelevant predictors in the regression model will tend to misleading inference. There are so many model selection methods available in the literature; among these, some methods are resistant to vertical outliers, but still, the problem of the presence of vertical outliers and leverage points is not well studied. In this article, we have modified the Sp statistic using the generalized M-estimator for robust model selection in the presence of vertical outliers and high leverage points. The proposed model selection criterion selects only relevant predictor variables by probability one for a large sample size. We found the equivalence of this criterion and the existing Cp and Sp criteria. The superiority of a proposed criterion is demonstrated using simulated and real data.

Development of a Wilks feature importance method with improved variable rankings for supporting hydrological inference and modelling

Abstract. Feature importance has been a popular approach for machine learning models to investigate the relative significance of model predictors. In this study, we developed a Wilks feature importance (WFI) method for hydrological inference. Compared with conventional feature importance methods such as permutation feature importance (PFI) and mean decrease impurity (MDI), the proposed WFI aims to provide more reliable variable rankings for hydrological inference. To achieve this, WFI measures the importance scores based on Wilks Λ (a test statistic that can be used to distinguish the differences between two or more groups of variables) throughout an inference tree. Compared with PFI and MDI methods, WFI does not rely on any performance measures to evaluate variable rankings, which can thus result in less biased criteria selection during the tree deduction process. The proposed WFI was tested by simulating monthly streamflows for 673 basins in the United States and applied to three interconnected irrigated watersheds located in the Yellow River basin, China, through concrete simulations for their daily streamflows. Our results indicated that the WFI could generate stable variable rankings in response to the reduction of irrelevant predictors. In addition, the WFI-selected predictors helped random forest (RF) achieve its optimum predictive accuracy, which indicates that the proposed WFI could identify more informative predictors than other feature importance measures.

Smooth-threshold multivariate genetic prediction incorporating gene-environment interactions.

We propose a genetic prediction modeling approach for genome-wide association study (GWAS) data that can include not only marginal gene effects but also gene–environment (GxE) interaction effects—i.e., multiplicative effects of environmental factors with genes rather than merely additive effects of each. The proposed approach is a straightforward extension of our previous multiple regression-based method, STMGP (smooth-threshold multivariate genetic prediction), with the new feature being that genome-wide test statistics from a GxE interaction analysis are used to weight the corresponding variants. We develop a simple univariate regression approximation to the GxE interaction effect that allows a direct fit of the STMGP framework without modification. The sparse nature of our model automatically removes irrelevant predictors (including variants and GxE combinations), and the model is able to simultaneously incorporate multiple environmental variables. Simulation studies to evaluate the proposed method in comparison with other modeling approaches demonstrate its superior performance under the presence of GxE interaction effects. We illustrate the usefulness of our prediction model through application to real GWAS data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI).

Abstract P114: Sparse Partial Least Squares Regression: A Promising Technique To Identify Heart-healthy Dietary Patterns In The Multi-Ethnic Study Of Atherosclerosis (MESA)

Most nutritional epidemiology studies investigating trends between diet and heart disease use outcome-independent dimension reduction methods, like principal component analysis, to create dietary patterns. While these methods construct patterns that describe important aspects of food consumption, these patterns are not inherently related to heart disease. Incorporating disease data into the pattern construction offers the possibility of more concisely summarizing the most disease-related foods. Sparse partial least squares (SPLS), one such method, was found to have favorable interpretation and prediction properties in the continuous outcome setting; while selecting a subset of relevant foods, it constructed a few dietary patterns that were correlated with BMI while also capturing variation in diet composition. These results were validated with simulated data. We propose incorporating SPLS into the Cox proportional hazards model to analyze a right-censored survival outcome. We hypothesized that this method would inherit the beneficial parsimony properties seen in the continuous setting, and we assessed whether this proposed method could use the most relevant covariates to create a few patterns that were associated with a survival outcome. While the proposed method targets covariate-level sparsity (i.e. variable selection), one competitor method exists that integrates pattern-level parsimony and partial least squares (PLS) in the Cox model, but it imposes more model parameters than the proposed method. We compared the variable selection, pattern selection, and predictive performance of four survival methods (Lasso, PLS, competitor sparse PLS, and proposed SPLS) via a simulation study. Simulation settings were informed in part by the Multi-Ethnic Study of Atherosclerosis (MESA), which has detailed food frequency questionnaire data on a large multi-ethnic population-based sample (6814 participants aged 45-84), as well as subsequent cardiovascular disease follow-up for over 15 years. In most studied simulation settings, the proposed method selected all 9 relevant predictors and the fewest number of irrelevant predictors (of 15) while creating a similar number of patterns and maintaining predictive ability of the outcome. In the setting most comparable to MESA, PLS chose all 24 predictors (by default) and 3.4 patterns (C-statistic=0.90), the competitor SPLS selected 21.1 predictors and 4.4 patterns (C-statistic=0.91), Lasso chose 16.4 predictors (C-statistic=0.91), and the proposed SPLS selected 11.7 predictors and 4.3 patterns (C-statistic=0.91), on average. We will also present an analysis of a coronary event in MESA using these four survival methods. In conclusion, we propose that using methods like SPLS to summarize food intake can create more heart disease-tailored dietary patterns that can complement the current nutritional epidemiology literature.

Improving the Accuracy of Subseasonal Forecasting of China Precipitation With a Machine Learning Approach

Precipitation change, which is closely related to drought and flood disasters in China, affects billions of people every year, and the demand for subseasonal forecasting of precipitation is even more urgent. Subseasonal forecasting, which is more difficult than weather forecasting, however, has remained as a blank area in meteorological service for a long period of time. To improve the accuracy of subseasonal forecasting of China precipitation, this work introduces the machine learning method proposed by Hwang et al. in 2019 to predict the precipitation in China 2–6 weeks in advance. The authors used a non-linear regression model called local linear regression together with multitask feature election (MultiLLR) model and chosen 21 meteorological elements as candidate predictors to integrate diverse meteorological observation data. This method automatically eliminates irrelevant predictors so as to establish the forecast equations using multitask feature selection process. The experiments demonstrate that the pressure and Madden–Julian Oscillation (MJO) are the most important physical factors. The average prediction skill is 0.11 during 2011–2016, and there are seasonal differences in forecasting skills, evidenced by higher forecast skills of winter and spring seasons than summer and autumn seasons. The proposed method can provide effective and indicative guidance for the subseasonal prediction of precipitation in China. By adding another three factors, Arctic Oscillation (AO) index, Western North Pacific Monsoon (WNPM) index and Western North Pacific Subtropical High (WNPSH) index into the MultiLLR model, the authors find that AO can improve the forecast skill of China precipitation to the maximum extent from 0.11 to 0.13, followed by WNPSH. Moreover, the ensemble skill of our model and CFSv2 is 0.16. This work shows that our subseasonal prediction of China precipitation should be benefited from the MultiLLR model.

Assessing the reliability of species distribution projections in climate change research

AbstractAimForecasting changes in species distribution under future scenarios is one of the most prolific areas of application for species distribution models (SDMs). However, no consensus yet exists on the reliability of such models for drawing conclusions on species’ distribution response to changing climate. In this study, we provide an overview of common modelling practices in the field and assess the reliability of model predictions using a virtual species approach.LocationGlobal.MethodsWe first review papers published between 2015 and 2019. Then, we use a virtual species approach and three commonly applied SDM algorithms (GLM, MaxEnt and random forest) to assess the estimated and actual predictive performance of models parameterized with different modelling settings and violations of modelling assumptions.ResultsMost SDM papers relied on single models (65%) and small samples (N < 50, 62%), used presence‐only data (85%), binarized models' output (74%) and used a split‐sample validation (94%). Our simulation reveals that the split‐sample validation tends to be over‐optimistic compared to the real performance, whereas spatial block validation provides a more honest estimate, except when datasets are environmentally biased. The binarization of predicted probabilities of presence reduces models’ predictive ability considerably. Sample size is one of the main predictors of the real model accuracy, but has little influence on estimated accuracy. Finally, the inclusion of ecologically irrelevant predictors and the violation of modelling assumptions increases estimated accuracy but decreases real accuracy of model projections, leading to biased estimates of range contraction and expansion.Main conclusionsOur ability to predict future species distribution is low on average, particularly when models’ predictions are binarized. A robust validation by spatially independent samples is required, but does not rule out inflation of model accuracy by assumption violation. Our findings call for caution in the application and interpretation of SDM projections under different climates.

A recursive ensemble-based feature selection for multi-output models to discover patterns among the soil nutrients

Finding the reduced and the relevant subsets of the predictors is inevitable when it comes to predictive modelling. If the datasets involved are heterogeneous and heteroscedastic in nature such as the soil samples, the task turns out to be trickier and such scenarios demand ensemble-based feature selection approach. The proposed algorithm uses an ensemble of filter (RReliefF), wrapper (Adaptive Plus –l and Minus –r), and embedded (Neighbourhood Component Analysis) approaches and applies it to the datasets in a heterogeneous and homogeneous manner. The Adaptive Plus –l and Minus –r is an experiment done on the Plus –l and Minus –r wrapper method to enhance the performance of the algorithm. The proposed combination rule of the ensemble filters out the irrelevant predictors for each response variable. Further, this ensemble is recursively implemented using the floating set of predictors to estimate the optimal subsets for multiple response variables at one go. Each step in recursion ensures that only the best subset of features (in terms of length and weights) among the current and previous iterations is retained. The Akaikes Information Criteria and the predictors weights calculated, assessed the efficiency of the resultant predictor sets of the recursive ensemble. The adjusted R2, Median Absolute Deviation and Root Mean Square Error on the unseen datasets confirmed the suitability of the same for predictive modelling in ecological domains.

Comparing the Lasso Predictor-Selection and Regression Method with Classical Approaches of Precipitation Bias Adjustment in Decadal Climate Predictions

AbstractIn this study, we investigate the technical application of the regularized regression method Lasso for identifying systematic biases in decadal precipitation predictions from a high-resolution regional climate model (CCLM) for Europe. The Lasso approach is quite novel in climatological research. We apply Lasso to observed precipitation and a large number of predictors related to precipitation derived from a training simulation, and transfer the trained Lasso regression model to a virtual forecast simulation for testing. Derived predictors from the model include local predictors at a given grid box and EOF predictors that describe large-scale patterns of variability for the same simulated variables. A major added value of the Lasso function is the variation of the so-called shrinkage factor and its ability in eliminating irrelevant predictors and avoiding overfitting. Among 18 different settings, an optimal shrinkage factor is identified that indicates a robust relationship between predictand and predictors. It turned out that large-scale patterns as represented by the EOF predictors outperform local predictors. The bias adjustment using the Lasso approach mainly improves the seasonal cycle of the precipitation prediction and, hence, improves the phase relationship and reduces the root-mean-square error between model prediction and observations. Another goal of the study pertains to the comparison of the Lasso performance with classical model output statistics and with a bivariate bias correction approach. In fact, Lasso is characterized by a similar and regionally higher skill than classical approaches of model bias correction. In addition, it is computationally less expensive. Therefore, we see a large potential for the application of the Lasso algorithm in a wider range of climatological applications when it comes to regression-based statistical transfer functions in statistical downscaling and model bias adjustment.

Development of a clinical model to predict secondary non-response of infliximab treatment in Crohn's disease.

Secondary non-response to infliximab (IFX) in patients with Crohn's disease (CD) is so common that prediction of non-response is required to adopt the optimal therapy for an individual patient. The clinical model was constructed in a training cohort. Clinical features, serological parameters, and genetic biomarkers at week 14 IFX therapy were obtained as predictors. Secondary non-response was defined at week 54. Univariate analysis was first performed to eliminate irrelevant predictors with non-response. Then, a logistic regression model was developed by using predictors with P < 0.1 in a univariate analysis. Finally, the model was validated with reference to its discrimination, calibration, and decision curve analysis (DCA). In a univariate analysis, erythrocyte sedimentation rate (ESR), the previous surgery, disease location, C-reactive protein (CRP), and TNFRSF1B (676 T>G) were found associated with secondary non-response to IFX. Logistic regression analysis with stepwise backward selection P < 0.05 then identified that ESR, the previous surgery, CRP, and TNFRSF1B (676 T>G) could serve as independent predictors, and a clinical model was developed based on the above predictors. Model 2 with TNFRSF1B (676 T>G) incorporated demonstrated more satisfactory discrimination (P = 0.029), better calibration (U P2 > 0.05), and higher clinical value in the validation cohort. The study presents a model to predict non-response to IFX in CD, which incorporates previous surgery, ESR, CRP, and TNFRSF1B (676 T>G). This model can be used to help clinicians adjust the therapeutic strategy and CD patients avoid unnecessary exposure to IFX.