Adaptive Variable Selection Research Articles

Accelerating tri-directional material distribution optimization in functionally graded plates with an adaptive design control point variable selection

In this paper, the problem of optimizing tri-directional material distribution in functionally graded (FG) plates under static loads to minimize the compliance is studied. Generalized shear deformation theory (GSDT) in the framework of isogeometric analysis is employed as the analyzer for efficiency and accuracy, and a non-uniform rational B-spline function is used for depicting the material distribution. As the number of control points in fine 3D design meshes, i.e., the number of optimization variables, can be too large for an algorithm to find a practically feasible design, we propose an adaptive variable selection mechanism. It gradually selects control points at important regions as variables by considering neighbor material variance, hence accelerates the optimization process. Various plate geometries are considered to validate the present approach. The findings confirm that the design mesh should be fine, and that the proposed variable selection strategy enables the optimization algorithm to find much more refined designs even in cases of large dimensions while also reducing computation.

Robust adaptive variable selection in ultra-high dimensional linear regression models

We consider the problem of simultaneous variable selection and parameter estimation in an ultra-high dimensional linear regression model. The adaptive penalty functions are used in this regard to achieve the oracle variable selection property with simpler assumptions and lesser computational burden. Noting the non-robust nature of the usual adaptive procedures (e.g. adaptive LASSO) based on the squared error loss function against data contamination, quite frequent with modern large-scale data sets (e.g. noisy gene expression data, spectra and spectral data), in this paper, we present a new adaptive regularization procedure using a robust loss function based on the density power divergence (DPD) measure under a general class of error distributions. We theoretically prove that the proposed adaptive DPD-LASSO estimator of the regression coefficients is highly robust, consistent, asymptotically normal and leads to robust oracle-consistent variable selection under easily verifiable assumptions. Numerical illustrations are provided for the mostly used normal and heavy-tailed error densities. Finally, the proposal is applied to analyse an interesting spectral dataset, in the field of chemometrics, regarding the electron-probe X-ray microanalysis (EPXMA) of archaeological glass vessels from the 16th and 17th centuries.

L0-Norm Variable Adaptive Selection for Geographically Weighted Regression Model

A geographically weighted regression (GWR) model with fewer explanatory variables and higher prediction accuracy is required in spatial analysis and other practical applications. This article proposes an -norm variable adaptive selection method to enhance performances of a GWR by simultaneously performing model selection and coefficient optimization. Specifically, we formulate a regularized GWR model with an additional -norm constraint to shrink those unimportant regression coefficients toward zero and propose an adaptive variable selection algorithm by iteratively distinguishing the important variables from the variable set. At each location, the best variable subset and optimizing coefficient estimations are simultaneously achieved under the -GWR framework. Moreover, two novel criteria, the modified Bayesian information criterion and the interpretability of coefficient symbol, which specify the variable selection and model interpretation, respectively, are also introduced to improve the performance of the -GWR. Experiments on both simulated and actual data sets demonstrate that the proposed algorithm can significantly improve the estimation accuracy of coefficients and can also enhance the interpretative ability of the established model.

Research on Aviation Safety Prediction Based on Variable Selection and LSTM.

Accurate prediction of aviation safety levels is significant for the efficient early warning and prevention of incidents. However, the causal mechanism and temporal character of aviation accidents are complex and not fully understood, which increases the operation cost of accurate aviation safety prediction. This paper adopts an innovative statistical method involving a least absolute shrinkage and selection operator (LASSO) and long short-term memory (LSTM). We compiled and calculated 138 monthly aviation insecure events collected from the Aviation Safety Reporting System (ASRS) and took minor accidents as the predictor. Firstly, this paper introduced the group variables and the weight matrix into LASSO to realize the adaptive variable selection. Furthermore, it took the selected variable into multistep stacked LSTM (MSSLSTM) to predict the monthly accidents in 2020. Finally, the proposed method was compared with multiple existing variable selection and prediction methods. The results demonstrate that the RMSE (root mean square error) of the MSSLSTM is reduced by 41.98%, compared with the original model; on the other hand, the key variable selected by the adaptive spare group lasso (ADSGL) can reduce the elapsed time by 42.67% (13 s). This shows that aviation safety prediction based on ADSGL and MSSLSTM can improve the prediction efficiency of the model while keeping excellent generalization ability and robustness.

Unobserved classes and extra variables in high-dimensional discriminant analysis

In supervised classification problems, the test set may contain data points belonging to classes not observed in the learning phase. Moreover, the same units in the test data may be measured on a set of additional variables recorded at a subsequent stage with respect to when the learning sample was collected. In this situation, the classifier built in the learning phase needs to adapt to handle potential unknown classes and the extra dimensions. We introduce a model-based discriminant approach, Dimension-Adaptive Mixture Discriminant Analysis (D-AMDA), which can detect unobserved classes and adapt to the increasing dimensionality. Model estimation is carried out via a full inductive approach based on an EM algorithm. The method is then embedded in a more general framework for adaptive variable selection and classification suitable for data of large dimensions. A simulation study and an artificial experiment related to classification of adulterated honey samples are used to validate the ability of the proposed framework to deal with complex situations.

Design of an Adaptive Nonnegative Garrote Algorithm for Multi-Layer Perceptron- Based Soft Sensor

In the study, we propose an adaptive variable selection algorithm for multi-layer perceptron (MLP)-based soft sensors. The proposed algorithm employs nonnegative garrote (NNG) to shrink the input weights of the trained MLP. To improve the shrinkage efficiency of the NNG, adaptive operators are designed using the mean impact value estimate. Moreover, the adaptive operators are data dependent, and are introduced into the constraints of NNG to make the shrinkage more efficient and effective. Cross-validation and Bayesian information criterion are used to determine the optimal garrote parameter for the NNG. The performance of the algorithm is validated using artificial datasets and a practical industrial application in coke dry quenching (CDQ) systems. The simulation results demonstrate that the adaptive mechanism improves the efficiency and precision of NNG, and has superior performance to other state-of-the-art algorithms. The result of variable selection is consistent with the experience of the field operator, and can provide technical supports for the optimisation and control of the CDQ process.

A fast algorithm for the accelerated failure time model with high-dimensional time-to-event data

We propose the logistic-kernel smoothing procedure for the semiparametric accelerated failure time (AFT) model with high-dimensional right-censored data. The resulting estimating procedure permits fast and accurate computation of regression parameter estimates and standard errors while preserving the same asymptotic properties as those from the non-smoothed rank estimating function. In addition, we provide an efficient numerical algorithm for obtaining a complete regularization path to facilitate adaptive variable selection in the AFT model. This can be done by using a second-order approximation of the smoothed estimating function and coordinate decent algorithm. Through extensive simulation studies, we examine several well-known penalties and show that our method is robust and computationally efficient with minimal loss of precision. Application to primary biliary cirrhosis (PBC) data demonstrates the utility of the proposed method in routine survival data analysis.

Bayesian group sequential enrichment designs based on adaptive regression of response and survival time on baseline biomarkers

Precision medicine relies on the idea that, for a particular targeted agent, only a subpopulation of patients is sensitive to it and thus may benefit from it therapeutically. In practice, it is often assumed based on preclinical data that a treatment-sensitive subpopulation is known, and moreover that the agent is substantively efficacious in that subpopulation. Due to important differences between preclinical settings and human biology, however, data from patients treated with a new targeted agent often show that one or both of these assumptions are false. This paper provides a Bayesian randomized group sequential enrichment design that compares an experimental treatment to a control based on survival time and uses early response as an ancillary outcome to assist with adaptive variable selection and enrichment. Initially, the design enrolls patients under broad eligibility criteria. At each interim decision, submodels for regression of response and survival time on a baseline covariate vector and treatment are fit; variable selection is used to identify a covariate subvector that characterizes treatment-sensitive patients and determines a personalized benefit index, and comparative superiority and futility decisions are made. Enrollment of each cohort is restricted to the most recent adaptively identified treatment-sensitive patients. Group sequential decision cutoffs are calibrated to control overall type I error and account for the adaptive enrollment restriction. The design provides a basis for precision medicine by identifying a treatment-sensitive subpopulation, if it exists, and determining whether the experimental treatment is superior to the control in that subpopulation. A simulation study shows that the proposed design reliably identifies a sensitive subpopulation, yields much higher generalized power compared to several existing enrichment designs and a conventional all-comers group sequential design, and isrobust.

Adaptive Variable Selection for Sequential Prediction in Multivariate Dynamic Models

We discuss Bayesian model uncertainty analysis and forecasting in sequential dynamic modeling of multivariate time series. The perspective is that of a decision-maker with a specific forecasting objective that guides thinking about relevant models. Based on formal Bayesian decision-theoretic reasoning, we develop a time-adaptive approach to exploring, weighting, combining and selecting models that differ in terms of predictive variables included. The adaptivity allows for changes in the sets of favored models over time, and is guided by the specific forecasting goals. A synthetic example illustrates how decision-guided variable selection differs from traditional Bayesian model uncertainty analysis and standard model averaging. An applied study in one motivating application of long-term macroeconomic forecasting highlights the utility of the new approach in terms of improving predictions as well as its ability to identify and interpret different sets of relevant models over time with respect to specific, defined forecasting goals.

Adaptive variable selection in nonparametric sparse additive models

We consider the problem of recovery of an unknown multivariate signal $f$ observed in a $d$-dimensional Gaussian white noise model of intensity $\varepsilon $. We assume that $f$ belongs to a class of smooth functions in $L_{2}([0,1]^{d})$ and has an additive sparse structure determined by the parameter $s$, the number of non-zero univariate components contributing to $f$. We are interested in the case when $d=d_{\varepsilon }\to \infty $ as $\varepsilon \to 0$ and the parameter $s$ stays “small” relative to $d$. With these assumptions, the recovery problem in hand becomes that of determining which sparse additive components are non-zero. Attempting to reconstruct most, but not all, non-zero components of $f$, we arrive at the problem of almost full variable selection in high-dimensional regression. For two different choices of a class of smooth functions, we establish conditions under which almost full variable selection is possible, and provide a procedure that achieves this goal. Our procedure is the best possible (in the asymptotically minimax sense) for selecting most non-zero components of $f$. Moreover, it is adaptive in the parameter $s$. In addition to that, we complement the findings of [17] by obtaining an adaptive exact selector for the class of infinitely-smooth functions. Our theoretical results are illustrated with numerical experiments.

A relative error-based approach for variable selection

The accelerated failure time model or the multiplicative regression model is well-suited to analyze data with positive responses. For the multiplicative regression model, the authors investigate an adaptive variable selection method via a relative error-based criterion and Lasso-type penalty with desired theoretical properties and computational convenience. With fixed or diverging number of variables in regression model, the resultant estimator achieves the oracle property. An alternating direction method of multipliers algorithm is proposed for computing the regularization paths effectively. A data-driven procedure based on the Bayesian information criterion is used to choose the tuning parameter. The finite-sample performance of the proposed method is examined via simulation studies. An application is illustrated with an analysis of one period of stock returns in Hong Kong Stock Exchange.

SLOPE-ADAPTIVE VARIABLE SELECTION VIA CONVEX OPTIMIZATION.

We introduce a new estimator for the vector of coefficients β in the linear model y = Xβ + z, where X has dimensions n × p with p possibly larger than n. SLOPE, short for Sorted L-One Penalized Estimation, is the solution to [Formula: see text]where λ1 ≥ λ2 ≥ … ≥ λ p ≥ 0 and [Formula: see text] are the decreasing absolute values of the entries of b. This is a convex program and we demonstrate a solution algorithm whose computational complexity is roughly comparable to that of classical ℓ1 procedures such as the Lasso. Here, the regularizer is a sorted ℓ1 norm, which penalizes the regression coefficients according to their rank: the higher the rank-that is, stronger the signal-the larger the penalty. This is similar to the Benjamini and Hochberg [J. Roy. Statist. Soc. Ser. B57 (1995) 289-300] procedure (BH) which compares more significant p-values with more stringent thresholds. One notable choice of the sequence {λ i } is given by the BH critical values [Formula: see text], where q ∈ (0, 1) and z(α) is the quantile of a standard normal distribution. SLOPE aims to provide finite sample guarantees on the selected model; of special interest is the false discovery rate (FDR), defined as the expected proportion of irrelevant regressors among all selected predictors. Under orthogonal designs, SLOPE with λBH provably controls FDR at level q. Moreover, it also appears to have appreciable inferential properties under more general designs X while having substantial power, as demonstrated in a series of experiments running on both simulated and real data.

ConvexLAR: An Extension of Least Angle Regression

The least angle regression (LAR) was proposed by Efron, Hastie, Johnstone and Tibshirani in the year 2004 for continuous model selection in linear regression. It is motivated by a geometric argument and tracks a path along which the predictors enter successively and the active predictors always maintain the same absolute correlation (angle) with the residual vector. Although it gains popularity quickly, its extensions seem rare compared to the penalty methods. In this expository article, we show that the powerful geometric idea of LAR can be generalized in a fruitful way. We propose a ConvexLAR algorithm that works for any convex loss function and naturally extends to group selection and data adaptive variable selection. After simple modification, it also yields new exact path algorithms for certain penalty methods such as a convex loss function with lasso or group lasso penalty. Variable selection in recurrent event and panel count data analysis, Ada-Boost, and Gaussian graphical model is reconsidered from the ConvexLAR angle. Supplementary materials for this article are available online.

Soft-sensor development with adaptive variable selection using nonnegative garrote

In this study, a soft-sensor modeling algorithm with adaptive partial least squares nonnegative garrote is developed by incorporating nonstationary disturbance. The approach is capable of monitoring the stationary and nonstationary behaviors of the process dynamics. The procedure of adaptive variable selection ensures that a compact and robust input–output relation is obtained online. Hence, in addition to simply tracking prediction, the model can be used for the detection of structural model change and the emergence of disturbance. The advantages of the proposed method are demonstrated with a simulation example and two industrial applications to predict the temperature of a blast furnace hearth wall and to estimate impurity composition of a distillation column.

Variable Selection via Regression Trees in the Presence of Irrelevant Variables

Many tree algorithms have been developed for regression problems. Although they are regarded as good algorithms, most of them suffer from loss of prediction accuracy when there are many irrelevant variables and the number of predictors exceeds the number of observations. We propose the multistep regression tree with adaptive variable selection to handle this problem. The variable selection step and the fitting step comprise the multistep method. The multistep generalized unbiased interaction detection and estimation (GUIDE) with adaptive forward selection (fg) algorithm, as a variable selection tool, performs better than some of the well-known variable selection algorithms such as efficacy adaptive regression tube hunting (EARTH), FSR (false selection rate), LSCV (least squares cross-validation), and LASSO (least absolute shrinkage and selection operator) for the regression problem. The results based on simulation study show that fg outperforms other algorithms in terms of selection result and computation time. It generally selects the important variables correctly with relatively few irrelevant variables, which gives good prediction accuracy with less computation time.

Multi-Step Classification Trees

Many algorithms originated from decision trees have been developed for classification problems. Although they are regarded as good algorithms, most of them suffer from loss of prediction accuracy, namely high misclassification rates when there are many irrelevant variables. We propose multi-step classification trees with adaptive variable selection (the multi-step GUIDE classification tree (MG) and the multi-step CRUISE classification tree (MC) to handle this problem. The variable selection step and the fitting step comprise the multi-step method. We compare the performance of classification trees in the presence of irrelevant variables. MG and MC perform better than Random Forest and C4.5 with an extremely noisy dataset. Furthermore, the prediction accuracy of our proposed algorithm is relatively stable even when the number of irrelevant variables increases, while that of other algorithms worsens.

Adaptive soft sensor for fluidized bed quality: Applications to combustion of biomass

The use of challenging, heterogeneous fuels and multi-fuel combustion is characteristic for modern-day circulating fluidized bed (CFB) combustion. This increases the need for monitoring the performance of the process. However, long-term online measurement of different phenomena in CFB boilers is a challenging task, and the lack of measurements complicates direct monitoring of the process. For this reason there is a need for new methods like soft sensors, which could be used to support or compensate the direct measurements of bed diagnostics. This paper demonstrates an adaptive soft sensor that utilizes process data to estimate the quality of the bed by predicting the grain size distribution in the bottom ash of circulating fluidized bed boilers. The grain size fractions from the sieving of bottom ash are used as reference values for bed quality. Either linear or nonlinear regression can be used to train the model. Adaptive variable selection is performed before constructing the dynamic model, which utilizes the selected variables to estimate the different grain size fractions online. The new approach has been tested using process data and grain size distributions measured from three individual CFB boilers fired by biomass, and the results show that the adaptive approach provides a useful tool for bed diagnostics.

An Adaptive Method of Variable Selection in Regression

An adaptive variable selection procedure is proposed which uses an adaptive test along with a stepwise procedure to select variables for a multiple regression model. We compared this adaptive stepwise procedure to methods that use Akaike's information criterion, Schwartz's information criterion, and Sawa's information criterion. The simulation studies demonstrated that the adaptive stepwise method is more effective than the traditional variable selection methods if the error distribution is not normally distributed. If the error distribution is known to be normally distributed, the variable selection method based on Sawa's information criteria appears to be superior to the other methods. Unless the error distribution is known to be normally distributed, the adaptive stepwise method is recommended.

Clustering gene expression profile data by selective shrinkage

Clustering of gene expression profiles is a widely used approach for finding macroscopic data structure. A complication in such analyses is that not all genes are informative for forming clusters and different clusters might have different transcription regulation. Driven by these considerations, we present a novel two-stage clustering approach. The first stage identifies informative genes by adaptive variable selection using pseudo-samples modeled by a high dimensional multigroup ANOVA model. Variables are selected using a rescaled spike and slab Bayesian hierarchical model having a special selective shrinkage property. The second stage uses output from the first stage for clustering. We demonstrate why selective shrinkage occurs, and by extension, why it is useful for the clustering paradigm. We analyze a human gene atlas expression dataset where the question of interest is to look for tissue-specific transcription regulation and investigate whether tissues can be grouped together due to similar genomic control.