High-dimensional Regression Problems Research Articles

TSK Fuzzy System Optimization for High-Dimensional Regression Problems

TSK Fuzzy System Optimization for High-Dimensional Regression Problems

Elastic net-based high dimensional data selection for regression

High-dimensional feature selection is of particular interest to researchers. In some domains, such as microarray data, it is quite common for a group of highly correlated explanatory variables to be of equal importance for inclusion in the predictive model. This paper proposes a new hybrid feature selection approach that integrates feature screening based on Kendall’s tau and Elastic Net regularized regression (K-EN). K-EN as an approach that embeds the Elastic Net, has the advantage of the grouping effect, which automatically includes all the highly correlated variables in the group. The K-EN approach offers insightful solutions to high-dimensional regression problems and improves Elastic Net performance since the screening phase is preceded by a step that further reduces the number of explanatory variables by removing those that disagree with the target based on Kendall’s tau. The use of Kendall’s tau further enhances Elastic Net performance, as it is robust enough to handle heavy-tailed distributions, non-parametric models, outliers, and non-normal data with greater ease. K-EN is therefore a time-saving approach. The proposed algorithm is evaluated on four simulation scenarios and four publicly available datasets, including riboflavin, eyedata, Longley, and Boston Housing, and achieves 0.2528, 0.0098, 0.1007, and 0.4121 respectively as the Mean Squared Error (MSE). K-EN’s MSEs are the best compared to those achieved by the state-of-the-art approaches reviewed in this paper. In addition, K-EN selects up to 100% of relevant features when run on simulated data.

Topology structure optimization of evolutionary hierarchical fuzzy systems

In this paper, a new method of hierarchical fuzzy system modeling for high-dimensional regression problems is proposed, which is called multi-objective evolutionary hierarchical fuzzy regression system (MOEHFRS). Different from the existing hierarchical fuzzy systems with fixed topology, the proposed MOEHFRS improves the accuracy of the model, reduces the total number of rules, and eliminates unnecessary features by flexibly constructing topology. In the process of topology evolution, MOEHFRS can exchange and combine different sub-fuzzy systems to achieve the selection and reuse of important features, as well as the elimination of unnecessary features, which improves the diversity of topology and enables the model to be well applied to high-dimensional regression problems. Different combinations of sub-fuzzy systems will result in different performance and number of rules. A new multi-objective evolutionary optimization algorithm is proposed to simultaneously optimize the number of rules and the accuracy of the model, which achieves the balance between complexity and accuracy of MOEHFRS. The proposed method is validated on 13 real world regression datasets and compared with other 5 methods. The results show that MOEHFRS is effective and advanced in terms of accuracy, number of rules, retention of important features and universality in different regression problems.

Applying Monte Carlo Dropout to Quantify the Uncertainty of Skip Connection-Based Convolutional Neural Networks Optimized by Big Data

Although Deep Learning (DL) models have been introduced in various fields as effective prediction tools, they often do not care about uncertainty. This can be a barrier to their adoption in real-world applications. The current paper aims to apply and evaluate Monte Carlo (MC) dropout, a computationally efficient approach, to investigate the reliability of several skip connection-based Convolutional Neural Network (CNN) models while keeping their high accuracy. To do so, a high-dimensional regression problem is considered in the context of subterranean fluid flow modeling using 376,250 generated samples. The results demonstrate the effectiveness of MC dropout in terms of reliability with a Standard Deviation (SD) of 0.012–0.174, and of accuracy with a coefficient of determination (R2) of 0.7881–0.9584 and Mean Squared Error (MSE) of 0.0113–0.0508, respectively. The findings of this study may contribute to the distribution of pressure in the development of oil/gas fields.

Robust variable selection and estimation via adaptive elastic net S-estimators for linear regression

Heavy-tailed error distributions and predictors with anomalous values are ubiquitous in high-dimensional regression problems and can seriously jeopardize the validity of statistical analyses if not properly addressed. For more reliable variable selection and prediction under these adverse conditions, adaptive PENSE, a new robust regularized regression estimator, is proposed. Adaptive PENSE yields reliable variable selection and coefficient estimates even under aberrant contamination in the predictors or residuals. It is shown that the adaptive penalty leads to more robust and reliable variable selection than other penalties, particularly in the presence of gross outliers in the predictor space. It is further demonstrated that adaptive PENSE has strong variable selection properties and that it possesses the oracle property even under heavy-tailed errors and without the need to estimate the error scale. Numerical studies on simulated and real data sets highlight the superior finite-sample performance in a vast range of settings compared to other robust regularized estimators in the case of contaminated samples. An R package implementing a fast algorithm for computing the proposed method and additional simulation results are provided in the supplementary materials.

LARGE-SCALE MULTIVARIATE SPARSE REGRESSION WITH APPLICATIONS TO UK BIOBANK.

In high-dimensional regression problems, often a relatively small subset of the features are relevant for predicting the outcome, and methods that impose sparsity on the solution are popular. When multiple correlated outcomes are available (multitask), reduced rank regression is an effective way to borrow strength and capture latent structures that underlie the data. Our proposal is motivated by the UK Biobank population-based cohort study, where we are faced with large-scale, ultrahigh-dimensional features, and have access to a large number of outcomes (phenotypes)-lifestyle measures, biomarkers, and disease outcomes. We are hence led to fit sparse reduced-rank regression models, using computational strategies that allow us to scale to problems of this size. We use a scheme that alternates between solving the sparse regression problem and solving the reduced rank decomposition. For the sparse regression component we propose a scalable iterative algorithm based on adaptive screening that leverages the sparsity assumption and enables us to focus on solving much smaller subproblems. The full solution is reconstructed and tested via an optimality condition to make sure it is a valid solution for the original problem. We further extend the method to cope with practical issues, such as the inclusion of confounding variables and imputation of missing values among the phenotypes. Experiments on both synthetic data and the UK Biobank data demonstrate the effectiveness of the method and the algorithm. We present multiSnpnet package, available at http://github.com/junyangq/multiSnpnet that works on top of PLINK2 files, which we anticipate to be a valuable tool for generating polygenic risk scores from human genetic studies.

The Price of Competition: Effect Size Heterogeneity Matters in High Dimensions

In high-dimensional sparse regression, would increasing the signal-to-noise ratio while fixing the sparsity level always lead to better model selection? For high-dimensional sparse regression problems, surprisingly, in this paper we answer this question in the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">negative</i> in the regime of linear sparsity for the Lasso method, relying on a new concept we term <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">effect size heterogeneity</i> . Roughly speaking, a regression coefficient vector has high effect size heterogeneity if its nonzero entries have significantly different magnitudes. From the viewpoint of this new measure, we prove that the false and true positive rates achieve the optimal trade-off <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">uniformly</i> along the Lasso path when this measure is maximal in a certain sense, and the worst trade-off is achieved when it is minimal in the sense that all nonzero effect sizes are roughly equal. Moreover, we demonstrate that the first false selection occurs much earlier when effect size heterogeneity is minimal than when it is maximal. The underlying cause of these two phenomena is, metaphorically speaking, the “competition” among variables with effect sizes of the same magnitude in entering the model. Taken together, our findings suggest that effect size heterogeneity shall serve as an important complementary measure to the sparsity of regression coefficients in the analysis of high-dimensional regression problems. Our proofs use techniques from approximate message passing theory as well as a novel technique for estimating the rank of the first false variable.

Subsampling for partial least-squares regression via an influence function

Partial least squares (PLS) performs well for high-dimensional regression problems, where the number of predictors can far exceed the number of observations. Similar to many other supervised learning techniques, PLS was developed in the framework of empirical risk minimization, which typically assumes that the test and training data are drawn from the same distribution. Any violation of this assumption can deteriorate the PLS performance. Subsampling via an influence function is a recently developed and promising technique for addressing this problem. However, influence functions are only guaranteed to be accurate for sufficiently small changes to the model, limiting their application to small-scale datasets. To overcome this obstacle, a new form of the influence function for PLS is derived, and a framework of subsampling via an influence function for PLS is developed. Compared with the classic PLS and two other subsampling frameworks, the results on four simulation datasets and two real-world datasets illustrate the effectiveness of our method.

A generalized stability estimator based on inter-intrastability of subsets for high-dimensional feature selection

Feature selection is an important preprocessing step in high-dimensional regression and classification problems because it helps to avoid the effect of noise, redundant, and irrelevant features on model performance. A variety of methods for feature selection have been proposed in the literature. However, small perturbations in the training data may produce highly different feature subsets; this is known as instability. Evaluating the stability of feature selection approaches has grown in importance and popularity in recent years. This paper introduces a novel stability estimator for measuring the internal and external stability of features subsets chosen using various methods in random subsampling experiments. The proposed estimator evaluates the similarity of features within selected subset as well as measuring the variation with respect to the number of selected features between selected subsets in different subsampling experiments. Furthermore, the asymptotic normality of the proposed stability estimator for large number of subsamples is also established. Experiments are carried out on both simulated and real-world datasets; where results demonstrate the usefulness of the proposed stability estimator.

Deep Neural Fuzzy System Oriented toward High-Dimensional Data and Interpretable Artificial Intelligence

Fuzzy systems (FSs) are popular and interpretable machine learning methods, represented by the adaptive neuro-fuzzy inference system (ANFIS). However, they have difficulty dealing with high-dimensional data due to the curse of dimensionality. To effectively handle high-dimensional data and ensure optimal performance, this paper presents a deep neural fuzzy system (DNFS) based on the subtractive clustering-based ANFIS (SC-ANFIS). Inspired by deep learning, the SC-ANFIS is proposed and adopted as a submodule to construct the DNFS in a bottom-up way. Through the ensemble learning and hierarchical learning of submodules, DNFS can not only achieve faster convergence, but also complete the computation in a reasonable time with high accuracy and interpretability. By adjusting the deep structure and the parameters of the DNFS, the performance can be improved further. This paper also performed a profound study of the structure and the combination of the submodule inputs for the DNFS. Experimental results on five regression datasets with various dimensionality demonstrated that the proposed DNFS can not only solve the curse of dimensionality, but also achieve higher accuracy, less complexity, and better interpretability than previous FSs. The superiority of the DNFS is also validated over other recent algorithms especially when the dimensionality of the data is higher. Furthermore, the DNFS built with five inputs for each submodule and two inputs shared between adjacent submodules had the best performance. The performance of the DNFS can be improved by distributing the features with high correlation with the output to each submodule. Given the results of the current study, it is expected that the DNFS will be used to solve general high-dimensional regression problems efficiently with high accuracy and better interpretability.

Multivariate Gaussian RBF‐net for smooth function estimation and variable selection

AbstractNeural networks are routinely used for nonparametric regression modeling. The interest in these models is growing with ever‐increasing complexities in modern datasets. With modern technological advancements, the number of predictors frequently exceeds the sample size in many application areas. Thus, selecting important predictors from the huge pool is an extremely important task for judicious inference. This paper proposes a novel flexible class of single‐layer radial basis functions (RBF) networks. The proposed architecture can estimate smooth unknown regression functions and also perform variable selection. We primarily focus on Gaussian RBF‐net due to its attractive properties. The extensions to other choices of RBF are fairly straightforward. The proposed architecture is also shown to be effective in identifying relevant predictors in a low‐dimensional setting using the posterior samples without imposing any sparse estimation scheme. We develop an efficient Markov chain Monte Carlo algorithm to generate posterior samples of the parameters. We illustrate the proposed method's empirical efficacy through simulation experiments, both in high and low dimensional regression problems. The posterior contraction rate is established with respect to empirical distance assuming that the error variance is unknown, and the true function belongs to a Hölder ball. We illustrate our method in a Human Connectome Project dataset to predict vocabulary comprehension and to identify important edges of the structural connectome.

A partially proximal linearized alternating minimization method for finding Dantzig selectors

The Dantzig selector (DS) is an efficient estimator designed for high-dimensional linear regression problems, especially for the case where the number of samples n is much less than the dimension of features (or variables) p. In this paper, we first reformulate the underlying DS model as an unconstrained minimization problem of the sum of two nonsmooth convex functions and a smooth coupled function. Then by exploiting the structure of the resulting model, we propose a partially proximal linearized alternating minimization method (P-PLAM), whose two subproblems are easy enough with closed-form solutions. Another remarkable advantage is that P-PLAM requires only one starting point, which is potentially helpful for saving computing time. A series of computational experiments on synthetic and real-world data sets demonstrate that the proposed P-PLAM has promising numerical performance in the sense that P-PLAM can obtain higher quality solutions by taking less computing time than some existing state-of-the-art first-order solvers.

Group orthogonal greedy algorithm for change-point estimation of multivariate time series

This paper proposes a three-step method for detecting multiple structural breaks for piecewise stationary vector autoregressive processes. The number of structural breaks can be large and unknown with the locations of the breaks being different among different components. The proposed method is established via a link between a structural break problem and a high-dimensional regression problem. By means of this connection, a group orthogonal greedy algorithm, originated from the high-dimensional variable selection context, is developed for efficiently screening out potential break-points in the first step. A high-dimensional information criterion is proposed for consistent structural breaks estimation in the second step. In the third step, the information criterion further determines the specific components in which structural breaks occur. Monte Carlo experiments are conducted to demonstrate the finite sample performance, and applications to stock data are provided to illustrate the proposed method.

High-dimensional sign-constrained feature selection and grouping

In this paper, we propose a non-negative feature selection/feature grouping (nnFSG) method for general sign-constrained high-dimensional regression problems that allows regression coefficients to be disjointly homogeneous, with sparsity as a special case. To solve the resulting non-convex optimization problem, we provide an algorithm that incorporates the difference of convex programming, augmented Lagrange and coordinate descent methods. Furthermore, we show that the aforementioned nnFSG method recovers the oracle estimate consistently, and that the mean-squared errors are bounded. Additionally, we examine the performance of our method using finite sample simulations and applying it to a real protein mass spectrum dataset.

A hybrid fuzzy feature selection algorithm for high-dimensional regression problems: An mRMR-based framework

One of the most important factors affecting the interpretability of Fuzzy Rule-Based Systems (FRBSs) is the number of features used Indeed, the employment of a large number of features could be problematic for the components of an FRBS. In these cases,feature selection approaches can be employed to discover the most beneficial features. In this study, we present HFFS a Hybrid Fuzzy Feature Selection algorithm for high-dimensional regression problems. It benefits from both filter and wrapper methods. HFFS is composed of two main components: a filter-based Selector and a wrapper-based Modifier. Selector is an mRMR-based framework that sequentially selects the most informative features and adds them to a candidate subset. However, since these features may not result in an efficient FRBS, they are evaluated by Modifier, which fine-tunes them whenever their overall performance declines. This procedure is repeated until no further progress is possible within the candidate subset, acting as the stopping point for the algorithm. The effectiveness of HFFS was evaluated using twenty-eight real-world regression datasets and four different types of FRBSs. The experimental results and statistical tests confirmed that HFFS was able to improve the estimation accuracy compared to the filter methods and reduce the computation times compared to the wrapper methods.

Selection Consistency of Lasso-Based Procedures for Misspecified High-Dimensional Binary Model and Random Regressors.

We consider selection of random predictors for a high-dimensional regression problem with a binary response for a general loss function. An important special case is when the binary model is semi-parametric and the response function is misspecified under a parametric model fit. When the true response coincides with a postulated parametric response for a certain value of parameter, we obtain a common framework for parametric inference. Both cases of correct specification and misspecification are covered in this contribution. Variable selection for such a scenario aims at recovering the support of the minimizer of the associated risk with large probability. We propose a two-step selection Screening-Selection (SS) procedure which consists of screening and ordering predictors by Lasso method and then selecting the subset of predictors which minimizes the Generalized Information Criterion for the corresponding nested family of models. We prove consistency of the proposed selection method under conditions that allow for a much larger number of predictors than the number of observations. For the semi-parametric case when distribution of random predictors satisfies linear regressions condition, the true and the estimated parameters are collinear and their common support can be consistently identified. This partly explains robustness of selection procedures to the response function misspecification.

Control-based algorithms for high dimensional online learning

In the era of big data, the high-dimensional online learning problems require huge computing power. This paper proposes a novel approach for high-dimensional online learning. Two new algorithms are developed for online high-dimensional regression and classification problems, respectively. The problems are formulated as feedback control problems for some low dimensional systems. The novel learning algorithms are then developed via the control problems. Via an efficient polar decomposition, we derive the explicit solutions of the control problems, substantially reducing the corresponding computational complexity, especially for high dimensional large-scale data streams. Comparing with conventional methods, the new algorithm can achieve more robust and accurate performance with faster convergence. This paper demonstrates that optimal control can be an effective approach for developing high dimensional learning algorithms. We have also for the first time proposed a control-based robust algorithm for classification problems. Numerical results support our theory and illustrate the efficiency of our algorithm.

A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems

Currently the training of neural networks relies on data of comparable accuracy but in real applications only a very small set of high-fidelity data is available while inexpensive lower fidelity data may be plentiful. We propose a new composite neural network (NN) that can be trained based on multi-fidelity data. It is comprised of three NNs, with the first NN trained using the low-fidelity data and coupled to two high-fidelity NNs, one with activation functions and another one without, in order to discover and exploit nonlinear and linear correlations, respectively, between the low-fidelity and the high-fidelity data. We first demonstrate the accuracy of the new multi-fidelity NN for approximating some standard benchmark functions but also a 20-dimensional function that is not easy to approximate with other methods, e.g. Gaussian process regression. Subsequently, we extend the recently developed physics-informed neural networks (PINNs) to be trained with multi-fidelity data sets (MPINNs). MPINNs contain four fully-connected neural networks, where the first one approximates the low-fidelity data, while the second and third construct the correlation between the low- and high-fidelity data and produce the multi-fidelity approximation, which is then used in the last NN that encodes the partial differential equations (PDEs). Specifically, by decomposing the correlation into a linear and nonlinear part, the present model is capable of learning both the linear and complex nonlinear correlations between the low- and high-fidelity data adaptively. By training the MPINNs, we can: (1) obtain the correlation between the low- and high-fidelity data, (2) infer the quantities of interest based on a few scattered data, and (3) identify the unknown parameters in the PDEs. In particular, we employ the MPINNs to learn the hydraulic conductivity field for unsaturated flows as well as the reactive models for reactive transport. The results demonstrate that MPINNs can achieve relatively high accuracy based on a very small set of high-fidelity data. Despite the relatively low dimension and limited number of fidelities (two-fidelity levels) for the benchmark problems in the present study, the proposed model can be readily extended to very high-dimensional regression and classification problems involving multi-fidelity data.

Wide spectrum feature selection (WiSe) for regression model building

Developing predictive models from industrial datasets implies the consideration of many possible predictor variables (features). Using all available features for data-driven modelling is not recommended, as most of them are expected to be irrelevant and their inclusion in the model may compromise robustness and accuracy. In this work, we present, test and compare a new two-stage feature selection method called wide spectrum feature selection for regression (WiSe). In the first stage, a combination of efficient bivariate filters analyzes linear and non-linear association patterns between predictors and responses, screening out clearly noisy features. In the second stage, the reduced set of retained features is subject to further selection in the scope of the predictive methods considered, optimizing their predictive performance. Three simulated datasets and an industrial case illustrate the effectiveness and benefits of applying WiSe to support model development in a wide range of high-dimensional regression problems.

Predicting the internal model of a robotic system from its morphology

The estimation of the internal model of a robotic system results from the interaction of its morphology, sensors and actuators, with a particular environment. Model learning techniques, based on supervised machine learning, are widespread for determining the internal model. An important limitation of such approaches is that once a model has been learnt, it does not behave properly when the robot morphology is changed. From this it follows that there must exist a relationship between them. We propose a model for this correlation between the morphology and the internal model parameters, so that a new internal model can be predicted when the morphological parameters are modified. Different neural network architectures are proposed to address this high dimensional regression problem. A case study is analyzed in detail to illustrate and evaluate the performance of the approach, namely, a pan–tilt robot head executing saccadic movements. The best results are obtained for an architecture with parallel neural networks. Our results can be instrumental in state-of-the-art trends such as self-reconfigurable robots, reproducible research, cyber–physical robotic systems or cloud robotics, in which internal models would available as shared knowledge, so that robots with different morphologies can readily exhibit a particular behavior in a given environment.